Test pypi upload fails with 502 Bad Gateway error

[smithsp]

Describe the bug During a GitHub Action to release the latest version of aurora, the package fails to upload to the pypi test server with a 502 Bad Gateway error

Expected behavior The Action has worked fine in the past

To Reproduce https://github.com/fsciortino/Aurora/runs/4558019561?check_suite_focus=true

My Platform GitHub Actions

Your help is greatly appreciated. Mention @fsciortino .

[di]

Thanks for the report. This is fixed in #10497.

[smithsp]

Your quick help is much appreciated. 🎉

[fsciortino]

@di unfortunately it seems that there's still some issue: we still get the same error when trying to upload the aurorafusion package. https://github.com/fsciortino/Aurora/runs/4582247971?check_suite_focus=true

I tried to increase the package version number and create a new release via github, but the error is still HTTPError: 502 Bad Gateway from https://test.pypi.org/legacy/. Could you please have a look?

[di]

@fsciortino Unfortunately I'm not seeing the corresponding exception in our error tracker. Can you share the actual distribution file attempting to be uploaded with us here so I can debug?

[di]

@fsciortino Looks like this started failing more than 10 days ago, so it's not likely due to this particular issue: https://github.com/fsciortino/Aurora/runs/4468671514?check_suite_focus=true

[fsciortino]

@di thank you very much for looking into this. Yes, I've been seeing the same error for more than a week, and only 3 days ago @smithsp opened this issue because we couldn't see what could be going wrong on our side. We're trying to upload the package via https://github.com/fsciortino/Aurora/blob/master/.github/workflows/python-publish.yml

How can I get a "distribution file" and share it with you?

[di]

I would recommend building your project locally (python setup.py sdist) and trying to upload it manually (twine upload -r testpypi dist/aurorafusion-2.0.4.tar.gz) to confirm that it also fails, then attach it here or host it elsewhere and link to it, whatever is easiest.

[fsciortino]

I confirm that testpypi also fails when doing a manual upload. Here's the dist file:

aurorafusion-2.0.4.tar.gz

[ewjoachim]

I might be wrong, and it's certainly not a reason for crashing but it looks like you set a very long description, and no long_description. Did you mixup the 2 fields by any chance ?

[di]

I see this error locally:

$ twine upload --repository-url http://localhost/legacy/ aurorafusion-2.0.4.tar.gz
Uploading distributions to http://localhost/legacy/
Enter your username: di
Enter your password:
Uploading aurorafusion-2.0.4.tar.gz
100%|███████████████████████████████████████████| 11.5M/11.5M [00:05<00:00, 2.33MB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from http://localhost/legacy/
".. image:: https://badge.fury.io/py/aurorafusion.svg     :target: https://badge.fury.io/py/aurorafusion      .. image:: https://anaconda.org/conda-forge/aurorafusion/badges/version.svg        :target: https://anaconda.org/conda-forge/aurorafusion  .. image:: https://anaconda.org/conda-forge/aurorafusion/badges/latest_release_date.svg        :target: https://anaconda.org/conda-forge/aurorafusion  .. image:: https://anaconda.org/conda-forge/aurorafusion/badges/platforms.svg        :target: https://anaconda.org/conda-forge/aurorafusion  .. image:: https://anaconda.org/conda-forge/aurorafusion/badges/license.svg        :target: https://anaconda.org/conda-forge/aurorafusion  .. image:: https://anaconda.org/conda-forge/aurorafusion/badges/downloads.svg        :target: https://anaconda.org/conda-forge/aurorafusion  Aurora is a package to simulate heavy-ion transportm neutrals and radiation in magnetically-confined plasmas. It includes a 1.5D impurity transport forward model, thoroughly benchmarked with the widely-adopted STRAHL code. It also offers routines to analyze neutral states of hydrogen isotopes, both from the edge of fusion plasmas and from neutral beam injection. A simple interface to atomic data for fusion plasmas makes it a convenient tool for spectroscopy and integrated modeling. Aurora's code is mostly written in Python 3 and Fortran 90. An experimental Julia interface has also been added.   Documentation is available at https://aurora-fusion.readthedocs.io.   Development  -----------  The code is developed and maintained by F. Sciortino (MPI-IPP) in collaboration with T. Odstrcil (GA), D. Fajardo (MPI-IPP), A. Cavallaro (MIT) and R. Reksoatmodjo (W&M), with support from O. Linder (MPI-IPP), C. Johnson (U. Auburn), D. Stanczak (IPPLM) and S. Smith (GA). The STRAHL documentation provided by R.Dux (MPI-IPP) was extremely helpful to guide the initial development of Aurora.  New contributors are more than welcome! Please get in touch at francesco.sciortino-at-ipp.mpg.de or open a pull-request via Github.   Generally, we would appreciate if you could work with us to merge your features back into the main Aurora distribution if there is any chance that the changes that you made could be useful to others.   Installation ------------  Aurora can be installed from PyPI using      pip install aurorafusion --user      You can omit the `--user` flag if you have write-access to the default package directory on your system and wish to install there.  Installing via conda is also possible using      conda install -c conda-forge aurorafusion            Both the PyPI and conda installation are automatically updated at every package release. Note that the conda installation does not currently install dependencies on `omfit_classes`, which users may need to install via `pip` (see the `PyPI repo <https://pypi.org/project/omfit-classes/>`_).   To look at the code and contribute to the Aurora repository, it is recommended to install from source, by git-cloning the  `Aurora repo <https://github.com/fsciortino/aurora>`_ from Github. This will ensure that you can access the latest version of the tools.   For compilation after git-cloning, users can make use of the `setup.py` file, e.g. using       python setup.py -e .  or use the makefile in the package directory to build the Fortran code using       make clean; make     Note that the makefile will not install any of the dependencies, listed in the `requirements.txt` file in the main directory. You can use this file to quickly install dependencies within a Python virtual environment, or install each dependency one at a time.  The Julia version of the code is not built by default. If you have Julia installed on your system, you can do        make julia  from the main package directory. This will build a Julia `sysimage` to speed up access of Julia source code from Python, but it is not strictly necessary. See the documentation to read about interfacing Python and Julia.    Atomic data -----------  Aurora offers a simple interface to download, read, process and plot atomic data from the Atomic Data and Structure Analysis (ADAS) database, particularly through the OPEN-ADAS website: www.open-adas.ac.uk . ADAS data files can be fetched remotely and stored within the Aurora distribution directory, or users may choose to fetch ADAS files from a chosen, pre-existing directory by setting      export AURORA_ADAS_DIR=my_adas_directory      within their Linux environment (or analogous). If an ADAS files that is not available in AURORA_ADAS_DIR is requested by a user, Aurora attempts to download it and store it there. If you are using a public installation of Aurora and you do not have write-access to the directory where Aurora is installed, make sure to set AURORA_ADAS_DIR to a directory where you do have write-access before starting.  Several ADAS formats can currently be managed -- please see the docs. Please contact the authors to request and/or suggest expansions of current capabilities.    License -------  Aurora is distributed under the MIT License. The package is made open-source with the hope that this will speed up research on fusion energy and make further code development easier. However, we kindly ask that all users communicate to us their purposes, difficulties and successes with Aurora, so that we may support users as much as possible and grow the code further.    Citing Aurora -------------  Please see the `User Agreement <https://github.com/fsciortino/Aurora/blob/master/USER_AGREEMENT.txt>`_." is an invalid value for Summary. Error: Field cannot be longer than 512 characters. See https://packaging.python.org/specifications/core-metadata for more information.

Which seems like @ewjoachim is correct, but not sure why this results in a 502 from PyPI instead of a 400. Will reopen this to investigate (since this isn't due to #10497), but short-term changing description to long_description should fix this.

[di]

I think shortening/truncating the bad value in the error message here would probably help.

[ewdurbin]

hmmm, from logs it looks like the backend is returning a 400...

[MY.IP.ADDR.ESS] - ewdurbin [20/Dec/2021:18:33:27 +0000] "POST /legacy/ HTTP/1.1" 400 17342 "-" "twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.6"

I'll try to determine what's turning that into a 503...

[ewdurbin]

Definitely looks like it's the fact that we're trying to shovel so much data into the HTTP Response Status line, which is a header. I'm guessing some part of our stack is choking on that, so @di's suggestion is a great start.

[ewdurbin]

In our HTTP stack we have client => fastly => elb => nginx-ingress => ghosttunnel => gunicorn

I tracked this down to ghosttunnel which is where TLS is finally dropped and we communicate over unix domain sockets to gunicorn. So even though gunicorn sends back a 400, ghosttunnel chokes on the giant Status header and bubbles a 502 all the way back up.

[fsciortino]

Thanks a lot for looking into this. Indeed, just changing description to long_description did the job and I was able to release a new version (first on test-pypi, then pypi). https://pypi.org/project/aurorafusion/

It's not clear to me why any of this happened though. I didn't change long_description to description recently... and it worked the way it was for quite a long time (approximately a year, I think). You may be already aware of the fact that something on the pypi end might have changed -- if so, it's all good. If not, hopefully this serves as a useful flag.

Many thanks again for your help!

[ewdurbin]

@fsciortino it's hard to say precisely what broke, but I took a little bit of a look and stopped when I reached the point where I saw you were using setup from numpy.distutils.core. given the massive number of versions of numpy and setuptools that could have been involved... I'm not sure I can commit the time to look any closer. PyPI itself has only seen changes to our upload endpoint to support new wheel platforms/architectures, nothing to do with metadata.

[ewdurbin]

Confirming this issue is resolved:

$ twine upload --repository-url https://test.pypi.org/legacy/ aurorafusion-2.0.4.tar.gz 
Uploading distributions to https://test.pypi.org/legacy/
Enter your username: ewdurbin
Enter your password: 
Uploading aurorafusion-2.0.4.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5M/11.5M [00:01<00:00, 8.68MB/s]
<Response [400]>
NOTE: Try --verbose to see response content.
HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
'.. image:: https://badge.fury..../master/USER_AGREEMENT.txt>`_.' is an invalid value for Summary. Error: Field cannot be longer than 512 characters. See https://packaging.python.org/specifications/core-metadata for more information.

convenient that we validate the form before checking permissions :-D

[fsciortino]

@ewdurbin thanks for pointing out that usage of setup from numpy.distutils.core is not recommended. I remember that when I setup this package I was having trouble using only setuptools, because aurorafusion has some fortran90 code to compile and I could get the Extension part of setup.py to work differently than how you see it now. If you have a minute to show me what would be the correct/best way to do this, I would really appreciate it. But I realize I already got great help from you guys in this Issue, so no worries if this takes too much time.