Closed ptheywood closed 1 year ago
gh
doesn't provide an obvious way to get the lists of assets for a release.
However, the GET /repos/{owner}/{repo}/releases
REST API endpoint does.
I.e.
curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/FLAMEGPU/FLAMEGPU2/releases
Returns a very long json response, which is a list ([]
) of dictionaries ({}
), with one per release (probably some pagination eventatiually.
The release dictionary includes "tag_name"
fields along with other releasae metadata such as if it is a pre-release or not, or a draft, publication date etc, but most importantly assets
assets
is a list of objets, where each object importantly contains "url"
, "name"
and "label" fields (and
"size"` which may be of interest)
Using the url
and name
components (labels maybe in the future) this should provide everything that is needed to generate a html page which allows python pip installation.
This API endpoint returns 30
release by default, which can be configured using the per_page
parameter. The page
parameter can be configured to select which page to view (1
indexed`.
A 404
status response will be issued if not found.
Alternatively, the GET /repos/{owner}/{repo}/releases/tags/{tag}
endpoint exists, to get the release info for a specific tag (i.e. no pagination).
Ie.
curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/FLAMEGPU/FLAMEGPU2/releases/tags/v2.0.0-alpha.1
If this output was stored to tag.json
it can then be filtered using jq
to extract various components
# Get the URL of the release
cat tag.json | jq -c '.["url"]'
# Get the url, name and label of the 0th asset for the release
cat tag.json | jq -c '.["assets"] | .[0]["url", "name", "label"]'
This could be used in conjunction with another endpoint to get the list of releases?
As a quick POC to verify this does work with gh release asset urls:
A file pyflamegpu.html
containing:
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl">pyflamegpu_vis-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl">pyflamegpu-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl">pyflamegpu_vis-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl">pyflamegpu-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl</a><br>
Then from a system with python 3.9 installed:
python3 -m pip install -U pyflamegpu -f pyflamegpu.html
Looking in links: pyflamegpu.html
Collecting pyflamegpu
Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1%2Bcuda112-cp39-cp39-linux_x86_64.whl (145.2 MB)
|████████████████████████████████| 145.2 MB 18.2 MB/s
Installing collected packages: pyflamegpu
Successfully installed pyflamegpu-2.0.0a1+cuda112
or the vis wheel:
python3 -m pip install -U pyflamegpu_vis -f pyflamegpu.html
Looking in links: pyflamegpu.html
Collecting pyflamegpu_vis
Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1%2Bcuda112-cp39-cp39-linux_x86_64.whl (146.2 MB)
|████████████████████████████████| 146.2 MB 16.8 MB/s
Installing collected packages: pyflamegpu-vis
Successfully installed pyflamegpu-vis-2.0.0a1+cuda112
However, both wheels can be installed into the same venv at the same time as they have different names, even though they both provide the pyflamegpu
module, so whichever was installed later will be imported.
This is one of the downsides of providing vis and non vis wheels which import the same package name.
It might be better to produce a separate .html
file for pyflamegpu
and pyflamegpu_vis
to further distinguish these as separate (and ideally in the future stop providing pyflamegpu_vis`
python3 -m pip list
Package Version
-------------- ---------------
pip 20.3.4
pkg-resources 0.0.0
pyflamegpu 2.0.0a1+cuda112
pyflamegpu-vis 2.0.0a1+cuda112
setuptools 44.1.1
Or if the .html file only contained python 3.8 URIs, then the following error would occur (i.e. what will happen on non x86_64 machines, or python < 3.6, or python > 3.9:
python3 -m pip install -U pyflamegpu -f pyflamegpu.html
Looking in links: pyflamegpu.html
ERROR: Could not find a version that satisfies the requirement pyflamegpu
ERROR: No matching distribution found for pyflamegpu
Attaching a .html file to a github release as an atrifact is not served in the web browser when viewed, it is downloaded, so can't be used for a (hosted) wheelhouse, so would need to be gh pages. Conda still seems like the better solution overall, given our wheels are not strictly manylinux compliant without dlopen soup.
I've made a start on this, working in a standalone separate repository for now (so no one accidentally finds it and starts using it before it is "official")
https://github.com/ptheywood/FLAMEGPU2-wheelhouse
So far I have a script which will fetch upto the last 30 releases and all their artifcacts from teh github api into a json file.
Another script is then going to take that json file and dump out the appropriate HTML to disk.
CI will then be set up to do that, and commit the result to the gh-pages branch.
Once that all works, I'll make it more API friendly so it doesn't hit any rate limits etc, by only finding assets for releases it didn't already know about.
We can then either set the workflow to be triggered manually, or be triggered by workflow_dispatch, so once we publush a release (i.e. stop it beign a draft) it will be triggered.
I now have a pair of working scrtipts.
However, changes are needed for vis / non vis build selection, (and cuda selection?)
I.e. if I produce samples/pyflamepgu.html
currently, containing all wheels, just asking for pyflamegpu
gets 2.0.0rc+cuda112.vis
. Is this what we want as the default?
While how wheel houses work, its possible to provide a directory strucutre with multiple files.
This allows people to query teh full set for explicit versions / just get the default, or to alos have ways to state "give me the latest cuda 11.2" or "give me the latest vis build, regardless of cuda version".
Otherwise the python version comparison rules apply, so the latest cuda version, and then .vis is priority over no ".vis" component.
The alternative would be to tweak the build / python packaing and ship vis and non vis wheeels together, so one (or both) can be installed. This would need changes to import statements however (off the top of my head), or dlopen magic to make everything a vis build, but only link against the vis dependence if opted in to.
Now have CI working for single-file wheelhouse, currently hostest at https://ptheywood.uk/FLAMEGPU2-wheelhouse/ for demonstration purposes, but this will not be the final location.
(most likely wheelhouse.flamegpu.com
or something to that effect
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu
Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-rc/pyflamegpu-2.0.0rc0%2Bcuda112.vis-cp310-cp310-linux_x86_64.whl (170.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 170.5/170.5 MB 11.2 MB/s eta 0:00:00
Collecting astpretty
Using cached astpretty-3.0.0-py2.py3-none-any.whl (4.9 kB)
Installing collected packages: astpretty, pyflamegpu
Successfully installed astpretty-3.0.0 pyflamegpu-2.0.0rc0+cuda112.vis
It is also possible to install specific versions, getting the most "recent" build (i.e. cuda112, vis was specified differntly in alpha builds, which might need to change again in the next rc?)
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-a2
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu==v2.0.0-a2
Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.2/pyflamegpu-2.0.0a2%2Bcuda112-cp310-cp310-linux_x86_64.whl (153.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.6/153.6 MB 11.3 MB/s eta 0:00:00
Installing collected packages: pyflamegpu
Attempting uninstall: pyflamegpu
Found existing installation: pyflamegpu 2.0.0rc0+cuda112.vis
Uninstalling pyflamegpu-2.0.0rc0+cuda112.vis:
Successfully uninstalled pyflamegpu-2.0.0rc0+cuda112.vis
Successfully installed pyflamegpu-2.0.0a2+cuda112
Or the fully explicit version to get a different cuda / non vis build
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-rc0+cuda110
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu==v2.0.0-rc0+cuda110
Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-rc/pyflamegpu-2.0.0rc0%2Bcuda110-cp310-cp310-linux_x86_64.whl (80.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.3/80.3 MB 13.6 MB/s eta 0:00:00
Requirement already satisfied: astpretty in ./.venv-remote/lib/python3.10/site-packages (from pyflamegpu==v2.0.0-rc0+cuda110) (3.0.0)
Installing collected packages: pyflamegpu
Attempting uninstall: pyflamegpu
Found existing installation: pyflamegpu 2.0.0a2+cuda112
Uninstalling pyflamegpu-2.0.0a2+cuda112:
Successfully uninstalled pyflamegpu-2.0.0a2+cuda112
Successfully installed pyflamegpu-2.0.0rc0+cuda110
Notably this does solve the python selection process, but to avoid vis wheels for the tutorial we will want to separate the vis into it's own html file. Likely the same for the cuda version, as I'm not sure of a way to save "Give me the most recent build which has +cu110 as the build identifier" (I've not looked really though).
I.e. people will still need to be pretty specific, unless they just want latest cuda with vis.
If we have a tree like follows:
.
├── cu110
│ └── index.html
├── cu112
│ └── index.html
├── cu120
│ └── index.html
└── vis
├── cu110
│ └── index.html
├── cu112
│ └── index.html
└── cu120
└── index.html
then users can could use one of the following, depending what they want
# latest cuda, currenlty with vis, latest pyflamegpu
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu
# latest cuda, specific pyflamegpu, iplicitly latest cuda + vis currently (though I do want to fix that still
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-rc
# specific cuda, no vis
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/cu110 pyflamegpu
# specific cuda, vis
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/vis/cu110 pyflamegpu
Alternatively / additionally, we could adjust teh name of the non-vis wheels to include an additional local version identifier component so it is "newer" than the equivalent +cudaXXX.vis
A version with more segements will be "greater" than a version with less, hence .vis
wins. We would need to make it lexicographically greater than .vis to appear newer, so no-vis
would not be defualt, nor console. So we might need to identify this separately. It might be better to rename the entire python package in that case (so both can be installed, and can explicitly choose the correct one, but that is a separate but related issue)
The local versions we embed to our wheels are also not allowed on pypi / should not be used when publishing "upstream" projects, so we might need to find an alternate way to deal with this in that case.
Partially implemented the above split wheelhouses to demonstrate it works.
https://github.com/ptheywood/FLAMEGPU2-wheelhouse/pull/1
Can install
index.hmtml
, defualts to latest cuda with visNo way to get a vis or non-vis CUDA X build. With the current single pyflamegpu package that is vis or not vis that doesn't make sense IMO. Plus given local versions are not allowed in package indexes such as pypi, we would only be able to distribute vis or non vis builds that way. Ideally we should have a separate package imo, and then either:
pip install pyflamegpu[pyflamegpuvis]
would be how to do a vis install. Vis builds should produce a different python package name, pyflamegpuvis
or wahtever. users would then have to change their import statements, but it would be as simple as doing import pyflamegpuvis as pyflamegpu
and that would be the onyl difference. A common snippet that would load console if available otherwise load vis (or the reverse) would then be:
try:
import pyflamegpu
except ImportError:
import pyflamegpuvis as pyflamegpu
Both are grim in their own ways and a decent bit of effort though.
Paul has setup a CNAME record for whl.flamegpu.com
. Once DNS propagates, I'll get this factored in and move the repo across to the flame org.
re: triggering the workflow on publication of a new release, need to make a PR against the FLAMEGPU2 repo, with a new github action workflow, which on publication of a release uses the gh api to trigger the workflow_dispatch
(or repository_dispatch
event on the wheelhouse repo's CI workflow.
Could do this with CURL, or the gh cli (though this apparently doesn't return the id of the new run, so have to query again for new runs and wait for it to complete. Roughly the following (given the published event for a repo only fires once)
on:
release:
types: [published]
env:
GH_TOKEN: ${{ secrets.MY_TOKEN }}
...
steps:
- run: |
gh workflow run ci.yml -R flamegpu/FLAMEGPU2
sleep 5
gh run watch -R flamegpu/FLAMEGPU2 $(gh run list -R flamegpu/FLAMEGPU2 -w ci.yaml -L1 --json databaseId --jq .[0].databaseId)
There is effectivly a race in the above, so the resulting status might be a lie (if the ci was triggered for another reason after the first gh command.
Slight issue with using gh to trigger a workflow in another repo, the GH_TOKEN generated in the workflow doens't seem to have enough permissions, even when adding write all. Might need some kind of org/account level token instead.
If the workflow is in the same repo it works as expected.
Using a PAT with repo action write scope stored in a secret let the remote dispatch work.
However PATS have a default expiration of 30 days, but also get deactivated after a year, so if I create one with a very long duration it will expire if there isn't a new flamegpu2 release for a while.
Grand scheme this is OK though, as we will be able to manually trigger it in that case, and I can add a comment to the workflow that would trigger it stating it would need a new PAT creating if it fails prior to re-running the job (or manually triggering the workflow in the other repo)
If we are going to release on pypi, we will likely only be able to host
pyflamegpu
packages built with a single cuda architecture, which Ideally should be the oldest version with broad support at the time of release. I.e. CUDA 11.2 would be my preference, as this will work with CUDA 11 releases >= 11.2, so wide support. Alternatively CUDA 11.0 for the oldest support, but this does not have forwards compatibility.In some cases,we might want to provide multiple version. Ie CUDA 11.0 (collab) and 11.2 (broader support), as well as in the future CUDA 11.x and CUDA 12.x. In which case we use the python local package version number to embed this, i.e.
+cuda112
for cuda 11.2.We could support users installing these via pip, withtout them being uploaded to pypi by leveraging the github release artifacts, github actions, and github pages.
Users could then install a potential 2.0.0 release built with CUDA 11.2 via:
Or alternatively pre-release versions using the
--pre
flag--no-index
may also be useful in the above commands?This is how pytorch handle pip installation of alternate versions.
As with pip distribution in general, it has downsides re: alterante builds. In this case you cannot ask for the latest pyflamegpu built with CUDA XY, instead you must specify a specific minor version and the local version. Poetry / other python packaging might also not be able to resolve this if they do not support the -f flag for pip subpackages.
One way to implement this would be:
Some of the following commands may be useful to implemetn this.
Using the
gh
command line tool which is installed into actions by default, we can:get a list of the last
10
releases using:get a json object containing all of the .whl assets for a specific release tag using:
For more information, see comments on #605.