Please open source this library

[Juul]

Having an open source WiFi MAC on such an affordable device would be extremely useful for experimenters and researchers.

If you are worried about people disregarding the WiFi standard then that is already possible with other devices.

The existing open WiFi drivers for cheap routers such as the AR9xxx driver that exists for the Omega2 $5 OpenWRT-compatible micro-router is easily configurable in ways that are harmful, outside of the WiFi specification, and illegal in various countries.

Software Defined Radios allow complete control over the WiFi MAC and PHY and these devices are getting cheaper every year.

Many devices already speak other non-WiFi protocols on the same 2.4 GHz frequencies as WiFi and standards-compliant WiFi devices knows how to handle the interference.

If you open source the WiFi MAC then the rest of the world can help improve your firmware to make it the best microcontroller WiFi MAC available.

Given all of this I do not understand why you would keep the WiFi MAC closed.

[igrr]

Open sourcing the upper MAC layer is certainly on our roadmap.

This does require a fair amount of developer time though, because large codebase has to be refactored, cleaned up, and documented. At this point we see more benefit in spending developer resources on those many features missing from the ESP-IDF. As the ESP-IDF gets more feature complete, we will invest more time into MAC layer clean up and open sourcing.

[Juul]

Well that's great news! You don't really need to clean up and refactor before releasing. As the open source mantra goes: Release early, release often. Having something is so much better than having nothing :)

[ghost]

If you release what you have now, we'll figure it out. We can even help you out! The point of having an open codebase is to have a constructive, bidirectional relationship with your users instead of releasing software for mere consumption.

[Spritetm]

The problem mostly is that what we have now isn't designed to be released and has developed some ball-of-yarn qualities. We need time to separate the bits of the MAC that we can open-source from the lower layers that need to stay closed because they contain IP-related sensitive information. As you can see, well, everywhere else in esp-idf, we're absolutely not against having a constructive and bidirectional relationship with our users; it's one of the reasons you see a gazillion commits to the esp-idf master branch and not a closed release full of binaries every x time! We just ask you to be patient on this one, we know it's a highly requested feature but due to the nature of the thing, we can't just throw it out in the open, sorry.

[Juul]

What would it take to open source the entire thing? Could we crowdfund however much it would take to buy the IP-protected parts free? How much would that cost and with whom would we negotiate?

[Spritetm]

I can't say... I'll pass it on, see if that is an option.

[Spritetm]

Sorry, just got a response; unfortunately in this case throwing money at the problem isn't going to help.

[ESP32DE]

Dear espressif

I am now at same thinking.

i do not want further more time wait for fixing things in the lib. we are helpless if we build productive things on the way to LIBS that are outdated or not fixed.

please open the libs that we can develope on Open Source way.

best wishes rudi ;-)

[dobegor]

@igrr Sorry to bother you here, but can we get some status update on upper layer open sourcing process? This is essential for a production-grade systems built with ESP32. Thanks in advance.

[danielinux]

I am hearing rumors saying Espressif will never open source this library. That's a shame, especially because the company is just riding the open source wave to spread within the community.

All I can say, they only care about selling hardware along with their proprietary, closed-source, bad quality software.

All in all, great marketing play. Well done!

[igrr]

Folks, indeed the process is a long one, but we are taking steps in opening parts of these libraries. Since this issue was opened, significant portion of rtc library was released, and recently ets_timer implementation was opened as well. In the foreseeable future, some parts (such as the PHY library) are going to stay closed source. But the supplicant and eventually upper MAC layer will be opened. Thank you for your patience and understanding.

[kaofishy]

Does this apply to the Bluetooth part as well? Since BLE shares pretty much the same Phy as NRF24L01 it would be interesting to bypass the protocol stack to ease interoperability.

[flo90]

Hi @igrr. I am currently writing my master thesis about energy saving in IEEE802.11. Beside other hardware I want to explore possibilities of the ESP32. But to do so, I need access to the wpa supplicant e.g. to restore key material and skip handshaking after deep sleep. The already implemented modem sleep gets close to another feature I need, except that I want to turning the modem off and on by myself instead of periodically. Do you see any problems doing so and can you tell me and all other people the current status? Maybe it would be a nice Christmas present opening the parts you announced ;-). Thanks in advance!

[stern0m1]

Any updates from espressif? I would also love for the wifi stack to be open source.

[igrr]

@stern0m1 no specific updates at the moment, but some parts of the supplicant (WPA2 and WPS) are planned to be open sourced in release 3.1.

[stern0m1]

Thanks. Off topic, but would the esp-now framework allow me to capture action frames not originating from another esp8266? Is there any other way with the current sdk to capture action frames from a different device? Promiscuous mode doesn't capture enough of the frame...

[igrr]

No, esp-now is built upon vendor IE, so it doesn't let you receive any extra packets.

[stern0m1]

If esp-now wouldn't filter by vendor I would be able to receive a command from an iPhone with one click. Direct without establishing a network.

This is another example of the benefits of open sourcing...

I have spent many hours/days trying to figure out a way to receive a command on the esp8266 from an iPhone with one click.

Any ideas? Thanks

[FlorianMickler]

Is it possible to open things up enough to route the timings of the wlan-ap master beacons out in order to do some time synching on them? That would be awesome and a big feature. (Multi Room Audio, Spatially separated Microphone Arrays...)

[hargathor]

Any update on the open source release process ? Is this still in the roadmap of the 3.1 at least for the Wifi Part ?

[igrr]

Parts of WPA2 and WPS source code are planned to be open sourced in 3.1.

[akshar001]

@igrr thank you for everything may we know when will mac layer library will be open sourced?

[igrr]

Recently WPA2 and WPS code has been moved to IDF. Next we plan to move more of wpa_supplicant code into IDF, but the exact scope for the next step is not determined yet. I will update this issue once more concrete plans are in place.

[akshar001]

@igrr sure we will wait for your update on this issue now? thanks

[chrismerck]

After years of struggling with dodgy WiFi drivers, the embedded systems community finally has a vendor who is committed to delivering transparency and quality in their WiFi software! The open-sourcing that has already occurred (particularly in wpa_supplicant) made a huge difference in the confidence that product designers can put in ESP32. Keep it up!

[jult]

Yes, open source this, please. I've just gotten 4 of Gosund's 220V switches, it's using this lib internally. I don't trust it on my WLAN without locking it out and putting it in a VLAN. Source needs to be audited.

[dereks]

@igrr

I'm a ~24 year OSS veteran (and Amiga before that, you Guru!), and the ESP32 is a dream come true. I've worked on various closed-source vendor firmware code bases, and it is universally junk compared to FreeRTOS, etc. Like most of us here, I am deeply grateful for all the work you have already done. I'll be contributing back, too.

So I like and trust your whole team. Kind of like how Amazon, Apple, and the CIA trusted Supermicro...

https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

The world is waking up to the security hole that is black-box hardware and firmware.

In the foreseeable future, some parts (such as the PHY library) are going to stay closed source.

With great respect, I do not see how you can defend this position given everything you've done on the ARM side. Clearly you get it, so why this limitation? I've written an 802.15.4 stack. The code wasn't magic or special... just a little messy because the protocol itself was messy.

I mean, srsly. Why not just release it today, warts and all? Please :)

And thank you again for ESP32. It will be my go-to WiFi module for the foreseeable future...

[synnack]

How about the lower MAC layer? ESP32 doesn't honor data rate settings from the BSS (and blasts 11b rates all over a european 4 channel plan, disturbing neighboring channels at a very slow rate).. It also doesn't allow setting data rates in AP mode, causing further issues in Europe where 11b rates are a big no, because we use 4 channel-plans (1,5,9,13). Maybe just an option to drop 11b support altogether?

I'd love to patch it, though :)

[wgaylord]

Any updates on this? I am slightly interested in trying to figure out how to "abuse" the "sdr" like part of my ESP32's Wifi for use in Ham Radio.

[aknooh]

@igrr Any updates from Espressif regarding open sourcing the MAC? How much longer do we have to wait? The open-source community has voiced their request since 2016. It's 2019 now. If Espressif is planning on open-sourcing MAC, is there anything that we can do to expedite the process?

[igrr]

@aknooh At the moment we are working on open sourcing the supplicant. By the end of Q2 we should be able to release that part. The largest chunk of work here is related to cleaning up the interactions and interfaces between the MAC and the supplicant.

With regards to expediting this process, it depends. If you are using the ESP32 in a commercial project, tell your Espressif sales representative that open-source MAC layer is important for you. If you are familiar with Wi-Fi internals and don't mind moving to Shanghai, check out our jobs page. If none of the above applies, leave a comment here. All this won't help short term, I'm afraid, but will probably help long term.

[stern0m1]

Are there any plans to enable the esp8266 in promiscous mode to capture more then the first 128 or so bytes? Via opensourceing or otherwise?

On Thu, Feb 21, 2019, 3:57 AM Ivan Grokhotkov <notifications@github.com wrote:

@aknooh https://github.com/aknooh At the moment we are working on open sourcing the supplicant. By the end of Q2 we should be able to release that part. The largest chunk of work here is related to cleaning up the interactions and interfaces between the MAC and the supplicant.

With regards to expediting this process, it depends. If you are using the ESP32 in a commercial project, tell your Espressif sales representative that open-source MAC layer is important for you. If you are familiar with Wi-Fi internals and don't mind moving to Shanghai, check out our jobs page. If none of the above applies, leave a comment here. All this won't help short term, I'm afraid, but will probably help long term.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/esp32-wifi-lib/issues/2#issuecomment-465916339, or mute the thread https://github.com/notifications/unsubscribe-auth/ARLap6cUNxvp9DdjrfCtuatYmsZVvGbjks5vPl-QgaJpZM4LP3Vz .

[Matheus-Garbelini]

@stern0m1 unfortunatelly you're not going to find answers here. Look for the oficial esp8266 repository: https://github.com/esp8266/Arduino

Regarding your question, I'm not sure about this limitation on the latest SDK, but I'm using version 2.0.0 with unrestricted packet injection bug, you can download the files here: https://github.com/spacehuhn/esp8266_deauther/wiki/Installation

From my tests, ESP8266 can receive/send even ACKs with no problems in promiscuous mode.

[ThinkalVB]

Without the open sourced part; can I implement packet forwarding between AP-STA and AP-AP by looking into the destination IP address.

[akshar001]

@igrr mac later any update? Sorry for troubling you with this question again and again!

[igrr]

@akshar001 The source code of wpa_supplicant has been moved into IDF in https://github.com/espressif/esp-idf/commit/c1396830243b4c8fbea255a04d8a04ed2cb9f3d7, which will be part of IDF 4.0. I will post in this topic once we have concrete plans in place for the next part.

[uis246]

@igrr на хабре в одном из комментариев вы писали, что откроете phy при наличии тех. переводчиков и способных программистов. С первыми затрудняюсь, а вот прогеров - целый osmocom. Возможно кого-нибудь оттуда можно будет привлечь и некоторые могу знать китайский. Жду phy. Надеюсь, что на esp32 будет документации не меньше, чем на Calypso.

[wgaylord]

Any updates? Or is this still in like limbo.

[wgaylord]

No new information since June?

[AshUK]

We Want This, We'd Love This, We Need This ❤️❤️❤️

[uis246]

No news bad news

[ESP32DE]

How is the status now near 4 years after and near end of year 2020 ( btw welcome and good first steps for RTOSes )

[reporiot]

Open sourcing the upper MAC layer is certainly on our roadmap.

This does require a fair amount of developer time though, because large codebase has to be refactored, cleaned up, and documented. At this point we see more benefit in spending developer resources on those many features missing from the ESP-IDF. As the ESP-IDF gets more feature complete, we will invest more time into MAC layer clean up and open sourcing.

It is past the four years now. Also no one is responding anymore. Maybe the company is bankrupt or something? Or @danielinux was right and the rumors are finally confirmed now.

It would have been helpful in research, ... very unfortunate.

[chrismerck]

@reporiot Although I too would like to see more (all) of the Wi-Fi stack open-sourced (and I share your sentiment that we were led to expect updates on this), I really must take issue with the fear-mongering and negative tone of your post. Such posts may work at cross purposes, discouraging Espressif from spending valuable team-member time communicating with the community.

I'll give another perspective. My company does use ESP32 commercially at decent scale. Although in an ideal world we would have a fully open Wi-Fi stack, for our business the openness and documentation of the BLE side has been of greater interest. An open BLE stack can allow for development of customized accessories that push the envelope of BLE's capabilities... and so we've used what limited rapport with Espressif we have to push in that direction rather than Wi-Fi. I've been pleased to see good progress in making the Bluetooth capabilities far more useful and flexible.

On the Wi-Fi side, the commercial pressures (again from my perspective) are for 5 GHz support and for excellent compatibility with the latest Wi-Fi routers & "mesh" routers, which have caused havoc for many IoT products. An open stack would be good for security researchers and unlock other community benefits, but it's hard for me to connect the dots to a commercial value for my business at least. Any tweaks to the Wi-Fi stack could result in unforeseen compatibility issues which would be a major headache.

With all that said, I would respectfully request an update on open sourcing of the Wi-Fi stack. Is this still in active development, as was the case for wpa_supplicant? Or has this been definitively de-prioritized? The expectation needs to be recalibrated here.

[moefear85]

There is no logical reason to keep the mac layer closed source, except to limit what the esp32 can do, probably so that it doesn't outperform other more expensive and complex fully-closed solutions. So in the end it's a financial decision. I don't understand why they stubbornly hold on to this policy despite more than one competitor arising, such as nRF24, nRF51/52, RTL8210, etc... What is clear though, is that it is a concious decision, since espressif is a large company, able to hire an army of talented programmers, and esp-idf hardly counts as a huge project that requires an unfathomable amount of man-hours. So claiming not having enough programmers or programming time to get it done, is obviously just an excuse. I think the real reason is perhaps that the alternatives can enforce fair-use of the wireless channel, while allowing an arbitrary payload, whereas an esp32 does all the necessary enforcing, such as packet length, in software, meaning in theory, if open sourced, one can probably generate an infintely long packet, hence jam that channel indefinately long. I'd make a wild guess and say esp doesn't want the masses to have easy access to this kind of damage, irrespective of how many proper use-cases there are out there.

As for alternatives, unnfortunately those solutions, despite being much more energy efficient, are much more limited in their hardware performance elsewhere, meaning they would have to accompany an esp32, just for the sake of overcoming the esp32's wifi limitations, such as very slow speed and unreliability of transmissions. This adds unnecessary cost. Still, I got so frustrated I just recently did thorough research and found out about these alternatives, that also support FreeRTOS and have solid development frameworks. I already purchased a few test modules, waiting for them to arrive. I hope they replace the esp32 in most use cases.

Anyways, we shouldn't insult espressif for their stubborness. It's their free choice. What bewilders me, is that there is such demand for a free and open-source mac layer, yet nobody ever made an effort to replace it. ESP-Open-SDK hasn't been updated in years, and there was never an attempt to open source any of the libraries, only the toolchain.

Bottom line... I advise everybody to do what I'm doing, which is discovering the alternatives, and when I have the time, reverse engineer the closed portions of esp-idf. It's not that difficult, it requires some commitment. All the closed source wifi functions are small enough to fathom/analyze. There is a large quantity of them though, but most of them are just garbage and mostly irrelevant helper functions. The juicy parts are few. There is enough documentation online as to the ISA, and there is even a qemu emulator for xtensa.

[wgaylord]

I think its more of its not their code to open source sort of problem. They most likely Licensed it from what ever companies Wifi IP actually makes the ESP32 tick.

[uis246]

I think the real reason is perhaps that the alternatives can enforce fair-use of the wireless channel, while allowing an arbitrary payload, whereas an esp32 does all the necessary enforcing, such as packet length, in software, meaning in theory, if open sourced, one can probably generate an infintely long packet, hence jam that channel indefinately long. I'd make a wild guess and say esp doesn't want the masses to have easy access to this kind of damage, irrespective of how many proper use-cases there are out there.

Jamming is not a rocket science. And neither computer one. You even don't need to write signle line of code.

[synnack]

I think the real reason is perhaps that the alternatives can enforce fair-use of the wireless channel, while allowing an arbitrary payload, whereas an esp32 does all the necessary enforcing, such as packet length, in software, meaning in theory, if open sourced, one can probably generate an infintely long packet, hence jam that channel indefinately long. I'd make a wild guess and say esp doesn't want the masses to have easy access to this kind of damage, irrespective of how many proper use-cases there are out there.

Jamming is not a rocket science. And neither computer one. You even don't need to write signle line of code.

Actually, ESP is currently already a jammer. It associates with 11b management rates which are 22MHz wide on 20MHz 11g channels. It ignores the allowed data rates set by the AP. Open source will only make this better, not worse.

Intentionally jamming is of course very easy.. just remove the magnetron from a microwave and power it. Done.

[Matheus-Garbelini]

I think this was already mentioned before, but their radio IP is from Riviera Waves (RW). So even if Espressif wants, unfortunately the source code cannot be disclosed because of NDAs associated with IP usage for both Wi-Fi and Bluetooth radio cores. Depending on what package RW has initially delivered to Espressif, the MAC implementation is also locked. This sort of things is common with silicon IP vendors. Is just that nowadays we are messing more and more with IoT stacks and people may get angry with some things the more the semi. ecosystem is exposed.

The Wi-Fi MAC layer on the other hand is implemented using 802.11 from freebsd. If anyone wants to learn how esp implements mac simply take a look on this code here: http://web.mit.edu/freebsd/head/sys/net80211/ieee80211_sta.c The only problem is that the MAC handling and radio entry points are proprietary, so espressif probably cannot release their modified freebsd 802.11 without exposing some interface and workarounds around the RW Wi-Fi radio IP.

Reverse engineering is certainly an option. This is the case for my Braktooth ESP32 firmware, which I hooked into the BT Classic baseband implementation to intercept or inject BT packets. But for usage in qemu, much more radio register information is needed then simply discovering how the RX and TX call paths and descriptors work.

[uis246]

I think this was already mentioned before, but their radio IP is from Riviera Waves (RW). So even if Espressif wants, the source code cannot be disclosure because of NDAs associated with IP usage for both Wi-Fi and Bluetooth radio cores.

Sounds like "We want you to buy and we don't want you to use it".