Markdown long description on wheel rendered as restructured text

[pevogam]

Hi, the project description as seen in https://pypi.org/project/guibot/ does not render as markdown despite me setting the long_description_content_type='text/markdown' option. Interestingly, when I did the same upload in the test repository, it also didn't render as markdown initially but did so after forceful refreshing in about 30 minutes. The same however did not happen at the production PyPI repo. Perhaps a relevant problem with refreshing is that when I search for the package through the page search engine, it returns a result which keeps having old version https://pypi.org/search/?q=guibot (0.11.4) and not the newer (0.20.1) despite days passing as well.

[ewdurbin]

@pevogam it doesn't look like Description-Content-Type was set at upload time.

Can you confirm that this package was uploaded using setuptools>=38.6.0 and twine>=1.11.0? For more detail see https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

[pevogam]

Sure, I have already checked the versions as well. This is the setuptools version

Python 2.7.13 (default, May 10 2017, 20:04:28) 
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> print setuptools.__version__
39.0.1

and this is the twine version:

$ twine --version
twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.18.4, setuptools: 39.0.1,
requests-toolbelt: 0.8.0, tqdm: 4.19.8)

I uploaded a previous version package without the Description-Content-Type first but the last uploaded version has this value set. Besides this, I have tested the newer version on the test repo where the Description-Content-Type was never missed (from the very beginning) and it still displayed restructured text first but after some refreshing finally started displaying Markdown. The same did not happen with the production repo.

[ewdurbin]

The version 0.20.1 uploaded to upload.pypi.org at 2018-03-23 13:58:59.920985 does not have the Description-Content-Type set. I've checked and this same version is on test.pypi.org, and also does not have this field set.

It looks like you've removed previous releases, so I'm unable to look at the progression of values over previous versions.

[pevogam]

Is it possible that the setuptools module reuses some cache? I have made sure to erase anything in advance before repackaging. This is my setup.py:

#!/usr/bin/python

from setuptools import setup
from os import path

p = path.abspath(path.dirname(__file__))
with open(path.join(p, '../README.md')) as f:
    README = f.read()

setup(
    name='guibot',
    version='0.20.1',
    description='GUI automation tool',
    long_description=README,
    long_description_content_type='text/markdown',

    install_requires=[
        "Pillow",
        'numpy',
        'opencv-contrib-python',
    ],
    tests_require=[
        'PyQt4',
    ],

    url='http://guibot.org',
    maintainer='Intra2net',
    maintainer_email='opensource@intra2net.com',
    download_url='',

    packages=['guibot'],
    package_dir={'guibot': '../guibot'},

    classifiers=[
          'Development Status :: 3 - Alpha',
          'Environment :: Console',
          'Intended Audience :: End Users/Desktop',
          'Intended Audience :: Developers',
          'Intended Audience :: System Administrators',
          'License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)',
          'Operating System :: MacOS :: MacOS X',
          'Operating System :: Microsoft :: Windows',
          'Operating System :: POSIX',
          'Programming Language :: Python',
          'Programming Language :: Python :: 2.7',
          'Topic :: Desktop Environment',
          'Topic :: Multimedia :: Graphics',
          'Topic :: Scientific/Engineering :: Artificial Intelligence',
          'Topic :: Software Development :: Testing',
          'Topic :: Software Development :: Libraries :: Python Modules',
    ],
)

[ewdurbin]

@pevogam how are you uploading the distribution that setuptools creates?

[pevogam]

I am uploading to https://upload.pypi.org/legacy/ as I followed instructions in a migration tutorial so this could be to culprit? The official page for this doesn't contain an address for this instruction. If the address is wrong then I wonder why I still see the new package on the web page.

In fact, you can see that the required setting for markdown is available from the source distribution

[ewdurbin]

no, that's the correct endpoint. you are uploading with twine?

[pevogam]

Yes, as per the instructions from https://packaging.python.org/tutorials/distributing-packages/ I do

twine upload dist/*

[ewdurbin]

bizarre. I just tested locally and everything is working as anticipated.

$ twine upload -r local guibot-0.20.1.tar.gz 
Uploading distributions to http://localhost/legacy/
Uploading guibot-0.20.1.tar.gz
100%|█████████████████████████████████████████████████████████████████████████| 7.64k/7.64k [00:00<00:00, 83.7kB/s]
$ twine --version
twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.18.4, setuptools: 39.0.1,
requests-toolbelt: 0.8.0, tqdm: 4.19.8)

[pevogam]

Can you verify that the actually uploaded source distribution contains the required field? Could it be some cache on the production server that didn't respect this field compared to your local machine?

[ewdurbin]

@pevogam distributions are not modified and are not cached by the upload endpoints. we explicitly accept, store, and serve exactly what clients upload.

[pevogam]

So then you see that the uploaded version contains the field exactly as required? This means that the uploader (here twine) did its job in packaging and sending it to the production server. The only possible difference then must be at the server.

[ewdurbin]

@pevogam unfortunately that's not for sure the case. are you positive that you uploaded the distribution for 0.20.1 on 2018-03-23 13:59:01.211308 using twine==1.11.0?

I'd be interested to see if creating a new release 0.20.2 and making sure to upload with twine==1.11.0 resolves the issue.

[pevogam]

I am sure I used the correct version since I installed twine especially for this purpose (using Fedora and automatically pulling the latest available version). But if you would like, we can wait until the next revision and I will reopen this if the issue reappears.

Strangely as I said before, the markdown was also not rendered on the test pypi repo at first but rendered later on (in ~30 minutes). I uploaded to the production repo after this.

Feel free to close this issue for the time being if you think my suggestion is good to try.

[di]

This is because https://github.com/pypa/wheel/pull/232 hasn't been released yet: your wheel has the wrong metadata version, twine has uploaded your wheel before the source distribution, and PyPI only considers the metadata for the first distribution it sees for a release.

I mention this in https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi, basically you need to do:

$ twine upload dist/<filename>.tar.gz
$ twine upload dist/<filename>.whl

instead.

[pevogam]

I see. Although this still doesn't explain why the rendering would succeed on the test repo and not on the production repo. Perhaps a race condition where they get uploaded in the correct order? When pypa/wheel#232 is released will it fix this issue right away or do affected packages need an additional upload with incremented version?

[di]

It seems to me like the same issue exists on the test repo: https://test.pypi.org/project/guibot/

Since it's not possible to edit the incorrect wheel metadata after uploading to PyPI, you'd need to make a new release (by incrementing the version).

[di]

Closing this as it's not technically an issue with Warehouse, but feel free to comment here if you're still having issues.

[pevogam]

Yes, I suggested that we do the same as I want to test your solution but I cannot force an increment in our version just because of the specifics of your implementation. Since you don't allow for package update even when it was caused by technical difficulties in the system I will have to wait for our next revision to possibly update this issue on my side. Thanks!

[brainwane]

@pevogam I hope Test PyPI helps you test this!

[pevogam]

As I already mentioned above I ran it through the test repo first but it didn't help. Thank you though! Perhaps it would be good if this change in two vs one upload steps is documented on the official PyPI page about this rather than @di's personal web page, at least until there is a release about it.

[brainwane]

Now that wheel 0.31.0 is out, I've suggested we add guidance to https://packaging.python.org/guides/making-a-pypi-friendly-readme/ giving people those minimum version numbers to make description rendering work properly:

• pkginfo>=1.4.2
• twine>=1.11.0
• setuptools>=38.6.0
• wheel>=0.31.0

Then https://packaging.python.org/tutorials/distributing-packages/#readme-rst-readme-md could link to that.

[gtalarico]

I was having a similar issue, package kept getting rejected for invalid long_desc (was in markdown and text/markdown was specified in setup.py) running pip install --upgrade setuptools wheel solved it for me.

[greyli]

Having the same issue. I have all requirements updated:

$ pip install --upgrade setuptools wheel twine
Requirement already up-to-date: setuptools in c:\python27\lib\site-packages (39.2.0)
Requirement already up-to-date: wheel in c:\python27\lib\site-packages (0.31.1)
Requirement already up-to-date: twine in c:\python27\lib\site-packages (1.11.0)

setup.py:

from setuptools import setup

basedir = path.abspath(path.dirname(__file__))
with open(path.join(basedir, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

setup(
    ...
    long_description=long_description,
    long_description_content_type='text/markdown',
    ...
)

PKG_INFO:

Metadata-Version: 2.1
...
Description: # Flask-CKEditor
...
Description-Content-Type: text/markdown

Here is the error output when upload:

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading Flask_CKEditor-0.4-py2.py3-none-any.whl
100%|##########################################################################################################################################| 4.95M/4.95M [01:15<00:00, 22.7KB/s]
Uploading Flask-CKEditor-0.4.tar.gz
100%|##########################################################################################################################################| 3.70M/3.70M [00:55<00:00, 33.8KB/s]
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. fo
r url: https://upload.pypi.org/legacy/

Any idea? It seems that only the wheel file (.whl) can be uploaded successful.

[javatechy]

It works if your setuptools is upgraded to version 38.6.0 or newer

twine is upgraded to version 1.11.0 or newer

wheel is upgraded to version 0.31.0 or newer

[ivikramsahu]

It works if your setuptools is upgraded to version 38.6.0 or newer

twine is upgraded to version 1.11.0 or newer

wheel is upgraded to version 0.31.0 or newer

version ::
>>> import wheel
>>> print(wheel.__version__)
0.33.6
>>> print(twine.__version__)
2.0.0
>>> import setuptools
>>> print(setuptools.__version__)
41.6.0

Still getting this weird issues.

[vikram.sahu@NL587 pepipost-sdk-python]$ twine upload dist/*
Enter your username: DX-Team
Enter your password: 
Uploading distributions to https://upload.pypi.org/legacy/
Uploading pepipost-2.6.1-py3-none-any.whl
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36.7k/36.7k [00:01<00:00, 19.1kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for Url: https://upload.pypi.org/legacy/

can anyone help me? for the first release, I didn't face such issues

[jamadden]

@vikram-sahu If you're still having problems, please open an issue in https://github.com/pypa/pypi-support/issues/. There's not an issue template, but it will help to see exactly what you're trying to upload (i.e., attach your wheel file to the issue).

I'll note that commonly if the long_description truly isn't RST, this often means that the long_description_content_type wasn't set, or that it wasn't readable because of other problems in the metadata, most often extra newlines in fields that are supposed to be one line, such as description= or license=.

[ivikramsahu]

@vikram-sahu If you're still having problems, please open an issue in https://github.com/pypa/pypi-support/issues/. There's not an issue template, but it will help to see exactly what you're trying to upload (i.e., attach your wheel file to the issue).

I'll note that commonly if the long_description truly isn't RST, this often means that the long_description_content_type wasn't set, or that it wasn't readable because of other problems in the metadata, most often extra newlines in fields that are supposed to be one line, such as description= or license=.

Thanks @jamadden for helping with comment. so i have checked this whl file using twine check dist/* it is gving me the below error

Checking dist/pepipost-2.6.1-py3-none-any.whl: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 158: Warning: Block quote ends without a blank line; unexpected unindent.
  warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.
Checking dist/pepipost-2.6.1.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 158: Warning: Block quote ends without a blank line; unexpected unindent.
  warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.

I have been debugging with my Readme.md file ( made a standalone script and parse the file it is getting properly converted to RST format)

[jamadden]

@vikram-sahu If you're still having problems, please open an issue in https://github.com/pypa/pypi-support/issues/. There's not an issue template, but it will help to see exactly what you're trying to upload (i.e., attach your wheel file to the issue). Please stop replying to closed issues.