search

YosysHQ / apicula

Project Apicula 🐝: bitstream documentation for Gowin FPGAs
MIT License
462 stars 66 forks source link

use compressed pickle files #183

Closed pepijndevos closed 1 year ago

pepijndevos commented 1 year ago

Untested.

Hopefully should resolve our pypi compression problems.

yrabbit commented 1 year ago

module 'importlib.resources' has no attribute 'files' What's your version of Python?

Here it uses the files API, which should work on Python versions >=3.9. At least for me it works on 3.9.

david65536 commented 1 year ago

I thought I had VSCode set up to use either usr/bin/python3 which is 3.9.6, or usr/local/bin/python3 which is 3.11.4. I downloaded `oss-cad-suite-darwin-arm64-20230714.tgz from the releases directory which is latest and built today.

It looks like maybe the toolchain is using the above release's oss-cad-suite/lib/python3.8/ which contains oss-cad-suite/lib/python3.8/site-packages/Apycula-0.8.2a2.dev1+g8e31050-py3.8.egg/apycula/gowin_pack.py.

I unzipped the Apycula-0.8.2a2.dev1+g8e31050-py3.8.egg file and confirmed that the gowin_pack.py inside has the change in line 984 from 32f21948799cf55f09af2e6abf48df38c91f9360 (this PR):

with gzip.open(importlib.resources.files("apycula").joinpath(f"{device}.pickle"), 'rb') as f:
        db = pickle.load(f)

I will try to see if I can get my environment to use something >= Python 3.9, but in the meantime could it be that something newer than python3.8 should be packaged in the YosysHQ/oss-cad-suite-build/releases? Thanks for your help and will try to use Python 3.9 next.

david65536 commented 1 year ago

I ended up using a previous release (oss-cad-suite-darwin-arm64-20230615.tgz) which is working for now.

pepijndevos commented 1 year ago

Oh darn. We claim to support Python 3.6 so this breakage wasn't intentional. Conceptually I think supporting whatever reasonably stable Linux distros are currently shipping is good practice. It looks like most have been on 3.9 or higher for a good while now: https://distrowatch.com/dwres.php?resource=package-in-distro&pkg=Python

I would gladly accept a PR that restores backwards compatibility either way.

I'll also ping @mmicko to see how they feel about Python versioning in oss-cad-suite.

david65536 commented 1 year ago

Sounds good! I will wait to hear what @mmicko thinks.

mmicko commented 1 year ago

@pepijndevos @david65536 right now we are distributing 3.8, moving to 3.9 would require change in complete infrastructure (since base cross compile image is using 3.8) and also making a big patch for mingw build to move to 3.9. We will move forward eventually but not now :( If possible please create a patch making it backward compatible with at least 3.8. Thanks

pepijndevos commented 1 year ago

So open_binary is what we can use in pre-3.11 when it became deprecated, and files is what we can use post-3.9.

I think something like gzip.GzipFile(fileobj=importlib.resources.open_binary("apycula", f"{args.device}.pickle")) might do the trick, modulo context manager stuff.

Not sure if we should add a version check, or just use the deprecated method until it's actually gone, I'm fine either way.

@david65536 do you want to make a PR, or should I clear some time to fix this?

david65536 commented 1 year ago

Sure I can take a look and make a PR!

david65536 commented 1 year ago

Sorry for the delay on this - I have a putative fix but was having trouble testing due to the Apycula==0.8.2a2.dev1+g8e31050 library being packaged into an egg. The workaround seemed to be a bit involved (editing the easyinstall entry scripts and a resulting few further steps which I got blocked on) - it seems easiest to push a PR then download and test a build for oss-cad-suite derived from that PR. Is that something that can currently happen via CI?

I am having some issues pushing the PR but could be due to my familiarity with git. Currently I get remote: Permission to YosysHQ/apicula.git denied to david65536. when I try to run git push.

git remote -v
origin  https://github.com/YosysHQ/apicula.git (fetch)
origin  https://github.com/YosysHQ/apicula.git (push)
pepijndevos commented 1 year ago

What you'll want to do is in the top right here click fork, which will make a repo under your account. Git clone that fork.

You can indeed push to that fork, create a PR from that, and download the egg from artifacts.

You can also copy the pickle files into apycula folder and run pip install -e . which makes an editable installation from the current folder. I would recommend doing that in a virtualenv.

git clone yourfork
cd apicula
viertualenv env
source env/bin/activate
# copy pickle files into apycula
pip install -e .
git checkout -b py38
git commit -am "support python 3.8"
git push

On Mon, 31 Jul 2023, 00:14 david65536 @.***> wrote:

Sorry for the delay on this - I have a putative fix but was having trouble testing due to the Apycula==0.8.2a2.dev1+g8e31050 library being packaged into an egg. The workaround seemed to be a bit involved (editing the easyinstall entry scripts and a resulting few further steps which I got blocked on) - it seems easiest to push a PR then download and test a build for oss-cad-suite derived from that PR. Is that something that can currently happen via CI?

I am having some issues pushing the PR but could be due to my familiarity with git. Currently I get remote: Permission to YosysHQ/apicula.git denied to david65536. when I try to run git push.

git remote -v origin https://github.com/YosysHQ/apicula.git (fetch) origin https://github.com/YosysHQ/apicula.git (push)

— Reply to this email directly, view it on GitHub https://github.com/YosysHQ/apicula/pull/183#issuecomment-1657278283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJFIIRV5ECYJREXN5EM4DXS3MDJANCNFSM6AAAAAAZLQWFCE . You are receiving this because you were mentioned. Message ID: @.***>

david65536 commented 1 year ago

Thanks @pepijndevos ! I made a PR here and was able to test in the context of the oss-cad-suite release using an egg I created. Being pretty new to all of this how do I refer to GW1N-1.pickle, GW1N-9.pickle, GW1NS-2.pickle, GW1NZ-1.pickle, GW1N-4.pickle, GW1N-9C.pickle, GW1NS-4.pickle, GW2A-18.pickle if I don't want to call them "the pickle files"?

pepijndevos commented 1 year ago

We usually call them chip DBs

On Thu, 3 Aug 2023, 05:35 david65536 @.***> wrote:

Thanks @pepijndevos https://github.com/pepijndevos ! I made a PR here https://github.com/YosysHQ/apicula/pull/189 and was able to test in the context of the oss-cad-suite release using an egg I created. Being pretty new to all of this how do I refer to GW1N-1.pickle, GW1N-9.pickle, GW1NS-2.pickle, GW1NZ-1.pickle, GW1N-4.pickle, GW1N-9C.pickle, GW1NS-4.pickle, GW2A-18.pickle if I don't want to call them "the pickle files"?

— Reply to this email directly, view it on GitHub https://github.com/YosysHQ/apicula/pull/183#issuecomment-1663243543, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJFIN6WWCQ2ME2XIBN74LXTMMB5ANCNFSM6AAAAAAZLQWFCE . You are receiving this because you were mentioned.Message ID: @.***>