setuptools (39.1.0) doesn't set up Description-Content-Type correctly on Windows

[hsiaoyi0504]

Originally reported in pypa/warehouse#4341.

It looks like long_description is full of Windows-style carriage-return characters (^M) which seem to have confused twine/pkginfo about what is and what isn't part of long_description.

My project at PyPI is at: https://pypi.org/project/gff3tool/1.4.2.

Libraries used:

wheel==0.31.1
setuptools==39.1.0
twine==1.11.0

[di]

The actual issue here is that setuptools is writing out a PKG-INFO file with CRLF line endings (^M) on some lines, which confuses things downstream for pkginfo/twine.

[ajostergaard]

I have the same symptoms.

I am using Cygwin, @hsiaoyi0504, are you also using Cygwin?

@di is it possible to confirm that I have the exact same issue? My project is https://pypi.org/project/django-payments-redsys/0.3/

Update: figured out where my PKG-INFO was, no CRLF line endings so must be a different problem. Still, seems like a setuptools issue as twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.19.1, setuptools: 40.1.0, requests-toolbelt: 0.8.0, tqdm: 4.24.0) and long_description_content_type='text/markdown' in setup but no long_description_content_type in PKG-INFO.

[hsiaoyi0504]

@ajostergaard Sorry, I didn’t use Cygwin.

[ajostergaard]

@hsiaoyi0504 oh well - worth a try. ;)

[ajostergaard]

In case its useful to anybody happening along, my problem was that I was using Python 2.7 to run setup... It works fine using Python 3.6. :)

@di tagging you in case you want to add a note to your blog post on the subject.

[pganssle]

Sorry, I am unclear on this issue. Seems like @ajostergaard was having problems unrelated to CRLF, but is the CRLF thing still a problem? Can we close this?

I have tagged this as "Needs Repro" for now, if anyone can provide a minimal reproducing example it would probably help us decide what action to take.

[di]

Here's another case of this happening: https://github.com/pypa/warehouse/issues/3664#issuecomment-439508132

[hurlenko]

Had the same issue on Windows 10. twine check was throwing

The project's long_description has invalid markup which will not be rendered on PyPI. The following syntax errors were detected:
line 29: Warning: Inline substitution_reference start-string without end-string.

wheel, twine, setuptools where of the latest version. I believe the problem was that I used long_description=open('Readme.md').read(). After I converted the whole project to LF the problem disappeared and I was able to publish my package to PyPI (with correct markdown rendering).

[matham]

I've also had a similar issue when trying to upload a sdist to PyPI that was rejected and twine check said:

Checking dist/Kivy-2.0.0rc1.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 1: Severe: Unexpected section title or transition.

    ====
  warning: `long_description_content_type` missing.  defaulting to `text/x-rst`.

Even though we did set long_description_content_type='text/markdown'. However, I could only reproduce it when the sdist was generated on GitHub Action on a Windows CI instance, not locally.

Specifically, we set the long description by doing:

with open(join(dirname(__file__), 'README.md'), 'rb') as fileh:
    long_description = fileh.read().decode("utf8")

But the generated Windows CI generated sdist (with python setup.py sdist --formats=gztar) created a sdist whose PKG-INFO was malformed. Specifically, it's Description field containing the description replaced all the \r\n in the original description with \r\r\n.

Changing our long_description to fileh.read().decode("utf8").replace('\r\n', '\n') fixed the issue.