CircuitPython Support for Package Resource Files

shulltronics commented 2 years ago

Background

Hi, I'm working on a package called unigui that wraps displayio functionality into a widget set for use on CircuitPython boards as well as general-purpose computers and single-board computers via Blinka.

Problem

In addition to terminalio.FONT, I would like to have other built-in fonts distributed with my package. I'm accomplishing this for CPython by including BDF font files in my package and using the pkgutil module in the standard library to get their path and load them at runtime using adafruit_bitmap_font.load_font. See this for examples. However, this solution relies on a module that isn't available in CircuitPython.

Possible Solutions

Is there any way to reference static package resource files in CircuitPython? Ideally it would work in CPython as well, so that the package doesn't need any platform-dependent logic. One possible solution inspired by the above link might be to implement the pkgutil module in CircuitPython. Another might be to do the same thing as terminalio.FONT, where I somehow compile in the loaded font as a member of my module, accessible, for example, via unigui.FONT0. It seems like terminalio.FONT is compiled into the CircuitPython executable though, and I'm not sure how I would accomplish that for my package.

Moving Forward

If there already exists a solution for this problem, please let me know. Otherwise, if either of the above options seems feasible, I would be interested in working on them. I'm also open to other ideas for solutions. Thanks for the time and any guidance you can provide!

jepler commented 2 years ago

Hi, thanks for your query! This sounds like an interesting use case, and the code would almost certainly be welcome in the community bundle if you chose to contribute it.

Can you identify a small subset of pkgutil that will satisfy your use cases and be implementable on CircuitPython? Perhaps you can prototype it in unigui, similar to:

try:
    from pkgutil import get_data
except ImportError: # Running on CircuitPython
    def get_data(package, name): ...

The CPython docs (https://docs.python.org/3/library/pkgutil.html#pkgutil.get_data) have a sketch of an implementation but you'd have to reduce it to string operations rather than using path functions..:

d = os.path.dirname(sys.modules[package].__file__)
data = open(os.path.join(d, resource), 'rb').read()

However, 'get_data' may not be ideal, as it is based on reading the whole file content into RAM; rather, we want to read fonts with APIs that operate either on filenames or on open file objects (I didn't check). I don't see that the core has a way to get the package's data as a stream instead.

shulltronics commented 2 years ago

Thanks so much for your guidance. I'm going to work on this over the next few days and will post back here once I make some progress.

jepler commented 2 years ago

Looking forward to your progress! I will be out for a few weeks so if you run into more questions I hope someone else will be able to help you out.

shulltronics commented 2 years ago

Hi all,

Sorry it took me so long to get back to this, but I finally have a working prototype to access my package resources in CircuitPython. In my initial code I was actually using pkg_resources (I'm honestly not sure why, as it seems this is shipped with setuptools and is not a great solution). After a bit more research, it seems that using importlib.resources is the current best practice and is available in the CPython stdlib. So I changed my code to use the files function from that module.

Either way, the issue remained similar (how to get the package resources when running on CircuitPython) and here is how I got that working:

import os
try:
    from importlib.resources import files
except ImportError:
    print("Couln't import importlib.resources... must be on CircuitPython!\nDefining custom files function.")
    import sys
    def files(module_name):
        sep = os.sep
        module_path = sys.modules[module_name].__file__
        dir_tree = module_path.split(sep)
        resource_path = sep
        for dir in dir_tree[0:-1]:
            if len(dir) > 0:
                resource_path += (dir + sep)
        return resource_path

See here for the full module code.

I think my main question now is if this really deserves to be put into a library of its own. It would be nice to avoid the conditional import and have a CircuitPython version on importlib.resources (and maybe some other functionality from that module could be useful for CircuitPython? I'm not sure). A current issue is that my custom files function isn't identical to the importlib.resources one because it returns a str instead of a pathlib object (I think).

If you all could provide me a bit of feedback about if you think it's worth it to start work on a dedicated CircuitPython library for this, and how I might get around the above issue. Thanks!! :)

adafruit / circuitpython