ENOMEM on ussl.wrap_socket() running upip on ESP32

[MathijsNL]

Version info: MicroPython v2.11.0.16 on 2019-12-03; ESP32 module with ESP32 ESPIDF_SUPHASH_V4 := 310beae373446ceb9a4ad9b36b5428d7fdf2705f (Version mentioned in the Makefile)

Problem When trying to install a package with upip using an ESP32 it gives the following error:

>>> upip.install('picoweb')
Installing to: /lib/
Error installing 'picoweb': OSError(12,), packages may be partially installed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "upip.py", line 265, in install
  File "upip.py", line 253, in install
  File "upip.py", line 217, in install_pkg
  File "upip.py", line 195, in get_latest_url_simple
  File "upip.py", line 171, in url_open
  File "upip.py", line 151, in url_open
OSError: [Errno 12] ENOMEM

I am not sure what the error means, but in case it is memory related this is the output from gc.mem_free():

>>> gc.mem_free()
63680

I just recently switched from uPython to Pycopy so the issue might be on my side.

If you need any more information let me know.

[pfalcon]

Yes, ENOMEM is "not enough memory", but not on Python side, but on system side. If you look at:

File "upip.py", line 151, in url_open

It is:

s = ussl.wrap_socket(s, server_hostname=host)

I.e., there it creates an SSL socket. That may require allocating quite large buffers, up to 16KB, and that of contiguous memory. I don't use ESP32, so not sure how large buffer(s) are allocated for it. E.g. on ESP8266 buffer specifically set to a low value (4-5K) to increase the chance of successful SSL socket creation (but then amount of data transferred is usually limited to that amount).

>>> gc.mem_free()
63680

That shows amount of total free memory, here the size of the largest contiguous block would be important. A general suggestion for cases like this is "try to run upip immediately after reset, when memory isn't yet fragmented by other operations".

[MathijsNL]

Thanks for your reply.

My main struggle is that after uploading the regular MicroPython firmware (daily build) Upip works and this error doesn't show up at all.

After uploading the Pycopy firmware immediately after booting it can't install a single package because this error is always there.

Do you have any suggestions on how to figure this out? Could I have made an error compiling the firmware?

[pfalcon]

That's indeed a strange and sad situation. Pycopy is somewhat behind the upstream in ESP32-related changes, I'm pulling them in these days, hopefully that might resolve the issue. Otherwise, the main focus of Pycopy is development of language-level features, nor particular hardware platforms, I treat Unix port as the reference platform.

So, trying to reproduce the issue with Unix port would be the way to tell whether it's regression in Pycopy/upip, or "hardware-specific" issue with ESP32. Note that it's easy to mimic a small heap condition on Unix port, see pycopy --help.

A quick test shows that following works:

pycopy -X heapsize=8wK -m upip install -p . picoweb

(For cleanness of experiment, would of course need to build 32-bit unix version, the above is with default-nowadays 64-bit.)

[pfalcon]

Looks like that's it: https://github.com/pfalcon/pycopy/commit/cbf14fd8223aa7ad8d1de15dc2858ff7299c4c58

[MathijsNL]

Confirmed, it is working perfectly now. Can this be closed?

MicroPython v2.11.0.16-179-g979ec245b on 2019-12-06; ESP32 module with ESP32
Type "help()" for more information.
>>>
>>> import upip
>>> upip.install('picoweb')
Installing to: /lib/
Warning: pypi.org SSL certificate is not validated
Installing picoweb from https://files.pythonhosted.org/packages/27/47/5d70d23ecc778bff5fdae3ef2eb57d59aff4a260da1b90