`long_description_content_type` flag in setup.py seems to be ignored on upload.

[fake-name]

Describe the bug I have a python package on PyPi that just (as of the last ~4-5 days) now fails to upload, with a new error:

durr@rwpscrape /m/S/S/WebRequest> python3 setup.py sdist upload

[snip build process]

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/
Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

Ok, that's clear enough. My readme is in markdown, let's see how to fix this.

Going to https://packaging.python.org/tutorials/packaging-projects/#description, it states you can add long_description_content_type="text/markdown", to your setup.py file, and it'll be processed as markdown.

<does edits>....
<submits to PyPi again

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/

[snip build process]

Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

What?

Expected behavior I expect if I specify the readme is markdown, as the documentation says to do so, the readme should be treated as markdown.

To Reproduce Attempt to upload a package with a readme that is valid markdown, but not valid reStructuredText.

My Platform

• Python 3.5.2 x64
• Ubuntu 16.04 LTS
• setuptools-39.2.0 (I specifically checked that it's the most current version)
• readme_renderer 20.0

Actual project in question is here. The readme.md in question is here, and the setup.py is here

[fake-name]

Note. https://pypi.org/help/#description-content-type also claims you can use other formats (and specifically mentions markdown!), but after installing readme_renderer, local checking appears to fail (?):

durr@rwpscrape /m/S/S/WebRequest> python3 setup.py check -r -s
running check
warning: Check: Not checking long description content type 'text/markdown', this command only checks 'text/x-rst'.

error: Please correct your package.
durr@rwpscrape /m/S/S/WebRequest>

Is markdown support a feature that's not actually been implemented yet? Did the documentation get released too early or something?

[jamadden]

You might try using the latest version of twine to handle the upload. Also, more metadata is available when you upload a wheel first, IIRC.

$ python setup.py sdist
$ pip install -U wheel
$ python setup.py bdist_wheel
$ twine upload dist/*whl dist/*gz

[jstaf]

@jamadden Nope, it's definitely broken with the latest twine/uploading a wheel first.

I think the issue is caused by a bug in https://github.com/pypa/readme_renderer - it's exiting with an error when it checks packages with a markdown readme (when it should not):

$ python setup.py check -s -r
running check
warning: Check: Not checking long description content type 'text/markdown', this command only checks 'text/x-rst'.

error: Please correct your package.

[theacodes]

I actually think this is due to #3980, I'll investigate now. Thank you for finding this!

[jstaf]

@theacodes - No worries! And yeah, that does seem more likely: it matches the timeframe of @fake-name 's original issue much more closely.

[theacodes]

@fake-name I can't reproduce this. See https://test.pypi.org/project/WebRequest/. I'm using setuptools 39.2.0 and twine 1.11.0.

If you wouldn't mind uploading your wheel and sdist somewhere else, I'd love to take a look.

[theacodes]

Also I see you tried using python3 setup.py sdist upload, please try with Twine.

[jstaf]

I think I found the issue - it's actually a problem with wheel. Spent a lot of time just now trying to make a reproducible example with one of my own repositories, only to find that my issues magically disappeared on creating a new Python environment and updating all my packages. It turns out that updating wheel from 0.30.0 to 0.31.0 fixes things (setuptools was 39.2.0 and twine was 1.11.0 in both cases).

[theacodes]

Gotcha. Makes sense. Shall we close this?

On Mon, May 28, 2018, 8:23 PM Jeff Stafford notifications@github.com wrote:

I think I found the issue - it's actually a problem with wheel. Spent a lot of time just now trying to make a reproducible example with one of my own repositories, only to find that my issues magically disappeared on creating a new Python environment and updating all my packages. It turns out that updating wheel from 0.30.0 to 0.31.0 fixes things (setuptools was 39.2.0 and twine was 1.11.0 in both cases).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pypa/warehouse/issues/4079#issuecomment-392642312, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPUc0qjc8sbdeqtJImtSwP8C8FHv90Qks5t3L87gaJpZM4UNuAe .

[jstaf]

Go for it! (also thanks for looking into this!)

[fake-name]

Submitting dist/WebRequest-0.0.29.tar.gz to https://upload.pypi.org/legacy/
Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

Still fails at my end. I manually updated wheel, setuptools was already up-to-date (Setuptools 39.2.0, wheel 0.31.1).

Twine is failing:

durr@rwpscrape /m/S/S/WebRequest> twine upload dist/WebRequest-0.0.29.tar.gz
Uploading distributions to https://pypi.python.org/pypi
UploadToDeprecatedPyPIDetected: You're trying to upload to the legacy PyPI site 'https://pypi.python.org/pypi'. Uploading to those sites is deprecated.
 The new sites are pypi.org and test.pypi.org. Try using https://upload.pypi.org/legacy/ (or https://test.pypi.org/legacy/) to upload your packages instead. These are the default URLs for Twine now.
 More at https://packaging.python.org/guides/migrating-to-pypi-org/ .

let me fix that, and see if it was relevant.

Ok, uploading with twine works, using setup.py does not. I've checked that setuptools, wheel, twine, and readme_renderer are all up to date, is there something I'm missing?

Some built packages:

SDist: WebRequest-0.0.30.tar.gz

BDist: WebRequest-0.0.30.linux-x86_64.tar.gz

[greyli]

Having the same issue. I have all requirements updated:

$ pip install --upgrade setuptools wheel twine
Requirement already up-to-date: setuptools in c:\python27\lib\site-packages (39.2.0)
Requirement already up-to-date: wheel in c:\python27\lib\site-packages (0.31.1)
Requirement already up-to-date: twine in c:\python27\lib\site-packages (1.11.0)

setup.py:

from setuptools import setup

basedir = path.abspath(path.dirname(__file__))
with open(path.join(basedir, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

setup(
    ...
    long_description=long_description,
    long_description_content_type='text/markdown',
    ...
)

PKG_INFO:

Metadata-Version: 2.1
...
Description: # Flask-CKEditor
...
Description-Content-Type: text/markdown

Here is the error output when upload:

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading Flask_CKEditor-0.4-py2.py3-none-any.whl
100%|##########################################################################################################################################| 4.95M/4.95M [01:15<00:00, 22.7KB/s]
Uploading Flask-CKEditor-0.4.tar.gz
100%|##########################################################################################################################################| 3.70M/3.70M [00:55<00:00, 33.8KB/s]
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. fo
r url: https://upload.pypi.org/legacy/

Any idea? It seems that only the wheel file (.whl) can be uploaded successful.

[theacodes]

Can you send over your sdist? I'd like to take a look at it.

[ocefpaf]

Just to provide another data point, Travis-CI auto deployment is also broken when using Markdown READMEs.

Here is the setup.py and as Travis-CI is using twine-1.11.0 and wheel-0.31.1, see https://travis-ci.org/pyoceans/erddapy/jobs/385360404#L1666.

The error message:

HTTPError: 400 Client Error: The description failed to render in the default format of
reStructuredText. See https://pypi.org/help/#description-content-type for more information.
for url: https://upload.pypi.org/legacy/ PyPI upload failed.

makes me think that something somewhere is ignoring that this is a Markdown file even though it is explicitly declared in setup.py as recommended in https://pypi.org/help/#description-content-type

[rpkilby]

long_description_content_type works with twine, but seems to have no effect when using setuptools.

Works:

• setup.py contains long_description_content_type='text/markdown',
• setup.py bdist_wheel (sdist also works)
• twine upload -r test dist/*

Does not work:

• setup.py contains long_description_content_type='text/markdown',
• setup.py bdist_wheel upload -r test

Pacakges:

pip==10.0.1
setuptools==39.2.0
twine==1.11.0
wheel==0.31.1

[ocefpaf]

@rpkilby based on your message I guess that Travis-CI is using setup.py bdist_wheel upload :-/

I'm trying to find the proper channel there to file an issue but it would be nice if setup.py sdist/bdist_wheel upload worked too until we all make the transition to twine.

[theacodes]

We're reverting this behavior in #4099, hang tight.

[frmdstryr]

A package I originally uploaded with no content type set, and then attempted to change to markdown by adding long_description_content_type="text/markdown" and it's also not working.

Is it possible to change the content type for an existing package?

[di]

Hi @frmdstryr, it's not possible to change anything about a published package -- they are essentially immutable.

It's likely that you have an out-of-date version of either twine, wheel or setuptools which cannot interpret this new metadata field. You can upgrade them all with:

$ pip install -U twine wheel setuptools

Once you do that, you can either increment the version number and make a new release, or make a post release to fix your description.

[frmdstryr]

Sorry, by existing I meant the package was originally uploaded without a content type set, it was a new release with an incremented version number.

Also setuptools and twine are already up to date

$ pip install --upgrade twine
Requirement already up-to-date: twine in /home/jrm/.local/lib/python2.7/site-packages
Requirement already up-to-date: requests!=2.15,!=2.16,>=2.5.0 in /usr/local/lib/python2.7/dist-packages (from twine)
Requirement already up-to-date: setuptools>=0.7.0 in /usr/local/lib/python2.7/dist-packages (from twine)

[di]

@frmdstryr Can you share a link to the release in question?

[frmdstryr]

https://pypi.org/project/enaml-native-cli/

[di]

@frmdstryr The wheel distribution for this release was built with wheel==0.29.0. You'll need the wheel to be built with wheel>=0.31.0 for it to be able to pick up this new metadata field.

[frmdstryr]

Ok, now it is working, thanks for the help. It appears just updating twine didn't also upgrade wheel.

[rpkilby]

Just to reiterate the current status of the issue - twine works, but setuptools still doesn't.

Has #4099 been deployed, or did the reversion not fix the issue?

[di]

@rpkilby Distutils doesn't support Metadata 2.1, so if you're uploading via python setup.py upload the long_description_content_type field won't be included.

[rpkilby]

@di Gotcha - I didn't realize that this wasn't handled by setuptools. Is there no way for setuptools to extend the metadata support provided by distutils?

edit: Also, what's the purpose of https://github.com/pypa/setuptools/pull/1286 if it doesn't support metadata 2.1?

[di]

@rpkilby Technically it might be possible, I'm not familiar enough with distutils/setuptools to say for sure that it would be. I suspect that it would likely require re-implementing large chunks of the upload command in setuptools.

However this is essentially what twine is (it's just not part of setuptools). So doing so would be a large duplication of effort for not a whole lot of benefit.

The PR at pypa/setuptools#1286 made it possible to create distribution artifacts with Metadata 2.1 metadata. On uploading, distutils/twine extracts this metadata from the distribution, and sends it along with the original file. PyPI doesn't do any introspection of the file, it depends solely on the metadata that is sent along.

Essentially, twine is able to extract all the metadata (including long_description_content_type) but distutils (invoked when using python setup.py upload) is not able to extract any fields from Metadata 2.1, so long_description_content_type never makes it to PyPI.

It might make more sense to deprecate uploading via setuptools instead. Seems like https://github.com/pypa/setuptools/issues/1381 is relevant here.

[mblayman]

I'm bummed that #3980 had to be reverted. 😢 I understand the reasons why and I think it was the right call.

In light of old clients not including the needed metadata, would we be open to fixing this if the process was more opt-in? The original solution ran the extra check if the there was any description data.

If I changed https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L895 to something like:

if form.description.data and form.description_content_type.data:

then the check would happen only if the content type was available too. This should permit older upload mechanisms to work (by skipping the check and permitting broken reST descriptions) and let users who want the extra checking to get that benefit too.

Would that be good enough or do we want to wait for clients to support the metadata format so that the check will be on by default for everyone?

[fake-name]

I wonder if it would be possible to do some heuristics, are there structures that are unique to markdown?

[theacodes]

I think one good option would be to make readme-renderer more gracefully fail, see https://github.com/pypa/readme_renderer/issues/102

[di]

Going to close this as the original issue was resolved, anyone still having an issue with Description-Content-Type should file a new issue.

[javatechy]

It works if your

setuptools is upgraded to version 38.6.0 or newer

twine is upgraded to version 1.11.0 or newer

wheel is upgraded to version 0.31.0 or newer

[tuanluu-agilityio]

Please update setuptools to latest version. It work fine for me. Thanks

[tidely]

I have setuptools upgraded to version 41.0.1 twine version 1.13.0 wheel version 0.33.4 and i am still getting this issue

[devfans]

Still have the issue with latest packages installed: setuptools 41.0.1
twine 1.13.0
wheel 0.33.4

HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

[devfans]

Seems I can make it working by modifying the README.md a bit, which is change ```python Some code ```

to ```

some code ```

[mit1280]

PyPI only renders rst(reStructuredText) file so if you have README.md then you have to convert into README.rst file after that you will be able to upload the file on PyPI.

Refer below site for install pandoc: [https://pandoc.org/installing.html]

run below command for convert file into .rst file python setup.py sdist pandoc --from=markdown --to=rst --output=README.rst README.md

check whether twine is able to render .rst file or not; twine check dist/*

Let me know if you encounter any other problems.

[jamadden]

PyPI only renders rst(reStructuredText) file so if you have README.md then you have to convert into README.rst file after that you will be able to upload the file on PyPI.

PyPI has supported markdown long descriptions since March 2018, so there's no need to convert.

[jagadeesh-l]

twine check dist/*

Checking dist\xx-1.1-py2.py3-none-any.whl: FAILED long_description has syntax errors in markup and would not be rendered on PyPI. line 1: Error: Document or section may not begin with a transition. warning: long_description_content_type missing. defaulting to text/x-rst.

[hlovatt]

I have same issue, e.g. like @jagadeesh-l describes, on version 3.2.0 of twine.

[stef]

i have the same issue. even after updating using pip install -U twine wheel setuptools and removing any language indicators after ```

[Syazvinski]

Im still having this issue as well. Anyone found a fix?

[phseiff]

I ran

sudo pip3 install --upgrade pip
sudo pip3 install --upgrade setuptools
sudo pip3 install --upgrade wheel
sudo pip3 install --upgrade twine
sudo pip3 install --upgrade setuptools_scm

and then, I ran

python3 setup.py sdist
twine upload -r testpypi -u <myusername> -p <mypassword> dist/*

and I still get

The description failed to render in the default format of reStructuredText. See https://test.pypi.org/help/#description-content-type for more information.

even though I correctly specified

    long_description=open('README.md', "r").read(),
    long_description_content_type="text/markdown",

in my setup.py.

It's also worth noting that uploading works just fine from my own machine, and used to work in GitHub actions too for the previous release, but I did not figure out where this difference in behavior comes from (probably because of different versions of Python modules).

The code in question is this GitHub actions pipeline (link points to the exact lines where the package is supposed to be pushed to PyPi for convenience).

Thanks in advance!

[di]

Anyone still having an issue with Description-Content-Type should file a new issue.