panda-re / panda

Platform for Architecture-Neutral Dynamic Analysis
https://panda.re
Other
2.45k stars 473 forks source link

Version disconnects between deb and python packages #1501

Open AndrewFasano opened 2 months ago

AndrewFasano commented 2 months ago

On each PR to main, we tag a new auto-version and push a .deb as a github release and a python package to pypi.

If the pypi package is identical to a prior release, pypi refuses to take the new release and we end up out of sync.

For example, after merging #1500 the release to pypi fails here: https://github.com/panda-re/panda/actions/runs/9357671824/job/25758919983.

Which means our latest version is now 1.8.28 for a .deb and 1.8.27 for the pypi release.

Consumers of our package can't use logic like this when these versions are out of sync:

ARG PANDA_VERSION="1.8.28"
ARG BASE_IMAGE="ubuntu:22.04"

FROM $BASE_IMAGE
# Download and install deb
RUN wget -O /tmp/pandare.deb https://github.com/panda-re/panda/releases/download/v${PANDA_VERSION}/pandare_$(echo "$BASE_IMAGE" | awk -F':' '{print $2}').deb && apt install -f *.deb

# Download and install panda pypi
RUN pip install pandare==${PANDA_VERSION}

I think it would be fine to have these versions completely disconnected or perfectly in sync. I don't think this current mostly-in-sync approach is good.

AndrewFasano commented 2 months ago

The root cause here might be different from what I initially identified, the publish python distribution job in the failed action (linked above) seems to reference version 1.8.27 which is the prior version, perhaps that's the cause of the failure. Since this was the job building a 1.8.28 release, I suspect that's wrong.

Checking dist/pandare-1.8.27-py3-none-any.whl: PASSED
Showing hash values of files to be uploaded:
/github/workspace/dist/pandare-1.8.27-py3-none-any.whl

SHA256: 30eae3f655c66c5eefb06413d69fca8d85adb3cbe46cb2b58e2d110ca9a947f5
MD5: a6d9fd77098cb7a21e28a68e55522e95
BLAKE2-256: 756ba7[16](https://github.com/panda-re/panda/actions/runs/9357671824/job/25758919983#step:4:17)ac5bc536b8d65b529bd821babdc94ddbdb3ed771dfdee38d760d793c

Uploading distributions to https://upload.pypi.org/legacy/
INFO     dist/pandare-1.8.27-py3-none-any.whl (676.3 KB)                        
INFO     password set by command options                                        
INFO     username: __token__                                                    
INFO     password: <hidden>                                                     
Uploading pandare-1.8.27-py3-none-any.whl
AndrewQuijano commented 1 month ago

Technically someone can get the correctly aligned PyPanda package from release too with the debian package https://github.com/panda-re/panda/releases/tag/v1.8.31

Based on history it seems aligned with 1.8.31, I can try hashing both pypanda files to see if the versions are identical https://pypi.org/project/pandare/#history

AndrewQuijano commented 1 month ago

Hm, this is weird, looks like for v1.30, v1.31 and v1.27, it works, and then just v1.28 pypi is blank, but it looks like it is aligned at the moment.

image

image

image

AndrewQuijano commented 2 weeks ago

While it seems nothing is broken at the moment exception to that one version, I think what we could do, is update publish_docker.yaml as follows:

1- Change create_release to run setup.sh twice, and store the debian packages as an artifact (much like what we do with the wheel file now). The packages are about 40 MB each, so given we have 2 GB of space, we should be fine, but can set a low retention period to be safe.

https://github.com/panda-re/panda/releases/tag/v1.8.34

2- In a separate job, download everything from create_release, then first thing first, attempt to publish to pip. If that succeeds, create the release and upload the debian and wheel file. This would enforce that is the pip fails, then no release is made.