search

esp-rs / esp-idf-sys

Bindings for ESP-IDF (Espressif's IoT Development Framework)
Apache License 2.0
264 stars 119 forks source link

cargo run / espflash flash... does not work offline. #334

Open mvasi90 opened 1 month ago

mvasi90 commented 1 month ago

Our systems (archlinux/gentoo) only have internet for certain applications/actions to their corresponding domains/subdomains and IP addresses.

The default policies drop both inbound, outbound and forwarding packets. (This should be the case for any enterprise and home computer to improve privacy and security).

We have been developing software in Rust for a long time and we understand that downloading libraries requires internet access. In such a case, access is allowed only to the cargo build instance and only to the domains and subdomains index.crates.io crates.io crates.io static.crates.io.

We are now creating projects in ESP32 (Xiao) and have encountered serious problems that prevent us from working securely and privately.

Neovim (as developing IDE) does not work.

image

Software development should certainly be offline.

developer@pc ~ $ sudo -g wireshark tshark --color -i lo -Y dns
[sudo] password for developer: 
Capturing on 'Loopback: lo'
    1 0.000000000    127.0.0.1 → 127.0.0.1    DNS 74 Standard query 0x38ca A api.github.com
    2 0.314967231    127.0.0.1 → 127.0.0.1    DNS 90 Standard query response 0x38ca A api.github.com A 140.82.121.6
    3 1.382931987    127.0.0.1 → 127.0.0.1    DNS 68 Standard query 0x9be3 A pypi.org
    4 1.383046958    127.0.0.1 → 127.0.0.1    DNS 132 Standard query response 0x9be3 A pypi.org A 151.101.64.223 A 151.101.192.223 A 151.101.128.223 A 151.101.0.223
    5 61.443101393    127.0.0.1 → 127.0.0.1    DNS 68 Standard query 0x66d4 A pypi.org

It tries to connect to github (Microsoft), pypi.org, etc.

Aug 10 21:21:18 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=140.82.121.6 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28011 DF PROTO=TCP SPT=47546 DPT=443 SEQ=3993256480 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A0A9D95970000000001030307) UID=1001 GID=1001 
Aug 10 21:21:19 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=140.82.121.6 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28012 DF PROTO=TCP SPT=47546 DPT=443 SEQ=3993256480 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A0A9D99830000000001030307) UID=1001 GID=1001 
Aug 10 21:21:19 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.64.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3360 DF PROTO=TCP SPT=36536 DPT=443 SEQ=575747781 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A73D5EFE70000000001030307) UID=1001 GID=1001 
Aug 10 21:21:20 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=140.82.121.6 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28013 DF PROTO=TCP SPT=47546 DPT=443 SEQ=3993256480 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A0A9D9D780000000001030307) UID=1001 GID=1001 
Aug 10 21:21:20 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.64.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3361 DF PROTO=TCP SPT=36536 DPT=443 SEQ=575747781 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A73D5F3D60000000001030307) UID=1001 GID=1001 
Aug 10 21:21:21 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=140.82.121.6 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28014 DF PROTO=TCP SPT=47546 DPT=443 SEQ=3993256480 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A0A9DA16E0000000001030307) UID=1001 GID=1001 
Aug 10 21:21:21 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.64.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3362 DF PROTO=TCP SPT=36536 DPT=443 SEQ=575747781 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A73D5F7C80000000001030307) UID=1001 GID=1001 
Aug 10 21:21:22 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=140.82.121.6 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28015 DF PROTO=TCP SPT=47546 DPT=443 SEQ=3993256480 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A0A9DA5630000000001030307) UID=1001 GID=1001
...

Cargo run (and espflash flash --monitor target/riscv32imac-esp-espidf/debug/...) does not work offline:

developer@pc ~/rust/blink $ cargo r
   Compiling esp-idf-sys v0.35.0
   Building [=======================> ] 205/213: esp-idf-sys(build)

Stuks here. And the network log is:

developer@pc ~ $ sudo -g wireshark tshark --color -i lo -Y dns
Capturing on 'Loopback: lo'
    1 0.000000000    127.0.0.1 → 127.0.0.1    DNS 68 Standard query 0xd025 A pypi.org
    2 0.000110841    127.0.0.1 → 127.0.0.1    DNS 132 Standard query response 0xd025 A pypi.org A 151.101.192.223 A 151.101.128.223 A 151.101.0.223 A 151.101.64.223
...
Aug 10 21:25:30 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32294 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CC76A0000000001030307) UID=1001 GID=1001 
Aug 10 21:25:31 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32295 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CCB580000000001030307) UID=1001 GID=1001 
Aug 10 21:25:32 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32296 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CCF4D0000000001030307) UID=1001 GID=1001 
Aug 10 21:25:33 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32297 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CD3430000000001030307) UID=1001 GID=1001 
Aug 10 21:25:34 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32298 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CD7380000000001030307) UID=1001 GID=1001 
Aug 10 21:25:35 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32299 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CDB2D0000000001030307) UID=1001 GID=1001 
Aug 10 21:25:37 pc kernel: out: IN= OUT=wlo1 SRC=192.168.0.200 DST=151.101.192.223 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32300 DF PROTO=TCP SPT=54420 DPT=443 SEQ=3038889709 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AE16CE3B80000000001030307) UID=1001 GID=1001
...

Whenever we are developing something on our device, should Microsoft be aware of it? Every time we are uploading a binary already compiled on our local device, should external servers know our IP address and other sensitive information?

Please fix that.

Vollbrecht commented 1 month ago

Well you can review the rustsite of the buildsystem in build.rs and the source code in embuild yourself. Github is run by microsoft and pulling esp-idf github repo is part of the process and that involves creating request against github. Also esp-idf heavily relight on python in it's cmake buildsystem and pull dependency's via pip. That all needs to be installed and managed somehow and potentially kept up to date. We try to make this process as friction-less as possible.

This friction-less setup might caught you off guard because you didn't review it yourself, but it helped countless people to even get a foothold into even starting this journey here. Keep in mind that this effort here is a community driven effort. We do it for the fun of it and if you want to influence or change something feel free to work on alternative means or create a PR improving documentation about it. If you need that for critical work stuff its your responsibility to keep things in order not the responsibility of some other folks on the internet.

Though in all earnestie i ask you the following question. Why are you ok with crates.io and not ok with pulling stuff from pypy via pip or pulling a git repo from github? I mean i understand that you may not expected that other services besides crates.io are involved, if you aren't aware how esp-idf works. But the argument to allow one but not the other fells of a bit short.

If that is not your cup of tea that is absolutely fine. There are a lot of ways to handle using esp-idf-sys even in an offline context. Though they require to learn more about the system and not just relight on others to do the heavy lifting, and some manual work.

So feel free disagree with what i said, though one point i want to clarify, if you are somewhat serious, before asking open-ended "rhetorical" questions about if Microsoft should be "aware" of what we are doing, consider reviewing the buildsystem as mention and feel free to drop a question, in the linked matrix channel, if you don't understand how a particular thing works.

mvasi90 commented 1 month ago

Why are you ok with crates.io and not ok with pulling stuff from pypy via pip or pulling a git repo from github?

Because crates.io is only used once when downloading the libraries. Not in each compilation (cargo build), much less in each execution (cargo run), much less in the development of the software.

It is absurd that every time I open the IDE to program, it is accessing the internet to see updates, and if it doesn't have internet access it doesn't allow me to program. I decide when I change the version of each library. And that is when I start cargo build to update.

Like crates.io we also have github.com and many other domains/subomains/IPs. But we don't tolerate internet access while we are programming.

I don't know your understanding of security and privacy, but free access to the internet should not exist if you want to maintain a secure environment.

There are a lot of ways to handle using esp-idf-sys even in an offline context. Though they require to learn more about the system and not just relight on others to do the heavy lifting, and some manual work.

Yes, we will have to investigate further the functionality of esp in rust. That requires more time and dedication to things outside of development, simply because esp in rust is designed to work online right out of the box.