Fix listing dependencies in json

Neradoc commented 2 years ago

By converting dependencies to _, the 1.10.2 release scrambled the requirements in the json file, moving all "dependencies" to "external dependencies" with an underscored pypi name instead of the library name (adafruit_circuitpython_hid instead of adafruit_hid).

This PR uses - not _ to reference modules from dependencies, so it matches the index used in package_list and the pypi_name field. Use the same function in both parts to emphasize their relatedness. The original name before transformation is still used in external_dependencies.

Note: this does not keep the same format as pipkin as intended by #90. pipkin can map the format function on the list to make sure every reference uses the same format without having to edit the list manually.

Additionally:

Add ~[; to filter requirements and strip() spaces. See here examples of valid formats.
Use sets to simplify the uniqueness test. Previously if line in dependencies would always be False. (testing the presence of adafruit-circuitpython-dotstar before adding adafruit_dotstar).
Add spidev to the list of skipped dependencies. (Found in the community bundle).
Sort the dependencies to make comparisons for that kind of check easier in the future. And use sort_keys in json, because why not.

With this PR the generated json matches the one generated by 1.10.1 except newly skipped libraries are skipped (pyasn1, pillow, spidev), and bad dependencies are fixed (typing-extensions~).

tekktrik commented 2 years ago

I can try to take a look at this tonight or tomorrow, but adding others in case they can get to it first!

Neradoc commented 2 years ago

I believe this bug has killed the project bundler's ability to include dependencies, since it's not looking in the right list. For example: https://learn.adafruit.com/smart-mirror-with-pyportal/coding-the-smart-mirror

lib
  adafruit_bitmap_font
  adafruit_display_text
  adafruit_pyportal

versus (installing adafruit_pyportal with circup)

lib
  adafruit_bitmap_font
  adafruit_bus_device
  adafruit_display_text
  adafruit_esp32spi
  adafruit_fakerequests.mpy
  adafruit_io
  adafruit_minimqtt
  adafruit_miniqr.mpy
  adafruit_pixelbuf.mpy
  adafruit_portalbase
  adafruit_pyportal
  adafruit_requests.mpy
  adafruit_touchscreen.mpy
  neopixel.mpy
  simpleio.mpy

adafruit / circuitpython-build-tools

Fix listing dependencies in json #93