search

pypa / readme_renderer

Safely render long_description/README files in Warehouse
Apache License 2.0
158 stars 88 forks source link

Don't allow using third-party lexers #168

Closed mroberge closed 2 years ago

mroberge commented 4 years ago

Describe the bug PyPI returned an error, "400 The description failed to render for 'text/x-rst'." Yet my package passes twine check dist/*

Expected behavior In the past, my README.rst file uploaded just fine. You can see it here https://pypi.org/project/HydroFunctions/

To Reproduce The package I am trying to upload is located in https://github.com/mroberge/hydrofunctions

python setup.py sdist bdist_wheel This produces "hydrofunctions-0.2.2.tar.gz" and "hydrofunctions-0.2.0-py3-none-any-whl".

(myactive_env) C:\mycomputer\PyDev\src\hydrofunctions>twine check dist/*
Checking dist\hydrofunctions-0.2.0-py3-none-any.whl: PASSED
Checking dist\hydrofunctions-0.2.0.tar.gz: PASSED

Everything seems fine and has passed so far.

twine upload dist/*

Things I've Tried

My final solution, which worked, was to create a simplified readme file, 'pypi_readme.rst' for upload as my long_description.

Somehow my .rst works for Sphinx, readthedocs, github, and ninjs.org, but not on PyPI.

My Platform

twine version: 3.1.1 wheel version: 0.34.2 setuptools version: 47.3.0 Python version: 3.7.1 Windows 10

Similar Issues

Additional context Full Error message:

C:\mycomputer\PyDev\src\hydrofunctions>twine upload dist/* --verbose

Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: mroberge
Enter your password:
Uploading hydrofunctions-0.2.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 346k/346k [00:01<00:00, 317kB/s]
Content received from server:
<html>
 <head>
  <title>400 The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information.</title>
 </head>
 <body>
  <h1>400 The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information.</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>
The description failed to render for &#x27;text/x-rst&#x27;. See https://pypi.org/help/#description-content-type for more information.

 </body>
</html>
HTTPError: 400 Client Error: The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/
di commented 4 years ago

Can you attach the distributions you're trying to upload here, or put them somewhere I can take a look at them?

mroberge commented 4 years ago

I went back to my earlier commit that failed to upload and re-built the dist.

I just figured out that you can see the "long_description" that gets generated by python setup.py sdist by looking in the dist/PKG-INFO file. This has a field, "description", which contains the rst text that PyPI uses as the homepage for each package. The text is padded with 8 spaces at the start of every line, but it is possible to remove those spaces and render it.

When I tried this, it passed both http://rst.ninjs.org/ and the distribution passed twine check dist/*

I've attached my sdist. Github won't take my wheel, but if it helps I can share it from a Google Drive or something.

Thank you!!! hydrofunctions-0.2.0.tar.gz

mroberge commented 4 years ago

==>Maybe PyPI shouldn't reject files due to rst errors. It seems like a good idea at first, but it doesn't fix the problem of projects with messed up homepages, and it creates new problems.

di commented 4 years ago

Here's what I get when I run twine check on that distribution:

$ twine check hydrofunctions-0.2.0.tar.gz
Checking hydrofunctions-0.2.0.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 57: Warning: Cannot analyze code. No Pygments lexer found for "ipython".
mroberge commented 4 years ago

I just created a new environment with Python 3.7, hydrofunctions, and twine. When I ran twine check on the distribution that I attached above, I didn't get that error.

Do you have any ideas for how I can reproduce this?

The error comes from a change I made in how I display example code in the rst file. The code that works uses the double colon syntax with the greater than signs, 

like this::
>>> python code example here

The file that doesn't work uses this notation:

        .. code-block:: ipython

            In  [3]: herring.ok
            Out [3]: True

I appreciate your help!

di commented 4 years ago

What version of twine do you have? twine --version

mroberge commented 4 years ago

Twine 3.1.1

di commented 4 years ago

There should be some way for users to see the rst file that is generated by the long_description in setup.py

The "file that is generated" is just whatever you pass as the description field in your setup.py, there's no additional manipulation that happens. The 8 spaces are just due to how setuptools serializes all the metadata.

There should be some way for users to see what errors occur when PyPI renders the long_description .rst

This is what twine check should be doing

Why doesn't twine check check the rst used by PyPI? or maybe it does, but it uses a different rendering engine?

I think this is the source of our issues 🙂

Even with PyPI checking and rejecting malformed rst, many projects have broken links and other rendering problems

Not sure what the suggestion is here, sorry!

di commented 4 years ago

OK, how about pygments and docutils?

$ pip show pygments
$ pip show docutils
mroberge commented 4 years ago

The environment that I used to look at the distribution I sent you has pygments 2.6.1 & docutils 0.16. When I filed this Issue I uploaded from an environment that I use for development that has pygments 2.6.1 & docutils 0.14

di commented 4 years ago

Hmm, it appears that IPython installs its own lexers:

https://github.com/ipython/ipython/blob/a2685d78f2403f84e6cb915ae11a4f6033ccc5f6/setup.py#L237-L241

Running twine check shouldn't take these into consideration. Moving this to pypa/readme_renderer as a bug.

mroberge commented 4 years ago

I figured out why my package failed on PyPI but not when I ran twine check locally.

My package requires Jupyter, which installs it's own lexer, apparently. If I run twine from the same environment that I build my distribution from, twine will pass my rst files. But if I run twine from a bare environment that hasn't installed my package, then twine will fail my distribution.

This will reproduce the issue::

(base) C:\mytest> conda create -n pypi-test python=3.7
(base) C:\mytest> activate pypi-test

(pypi-test) C:\mytest> pip install twine
(pypi-test) C:\mytest> twine check hydrofunctions-0.2.0.tar.gz
Checking hydrofunctions-0.2.0.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 57: Warning: Cannot analyze code. No Pygments lexer found for "ipython".

So, sure enough, I had updated my README.rst file from the last functioning version, to show code examples that look like they were run in Jupyter notebooks:

.. code-block:: ipython

    In  [3]: herring.ok
    Out [3]: True
miketheman commented 2 years ago

Took a look at this today, and it appears that passing ipython to code-block doesn't break the experience any longer, rather does not highlight at all, providing a regular <pre> block.

Using twine version 4.0.1, readme_renderer 35.0, I tested the sdist attached, as well as hydrofunctions 0.2.3 (latest release) and everything works with no warnings.

I'm going to assume this was fixed sometime in the past two years and recommend closure.

di commented 2 years ago

Closing for now but happy to revisit if this is still reproducible.

mroberge commented 2 years ago

Thank you for looking into this! As you say, ipython code-blocks don't get highlighted at all on pypi, but that is okay. The important thing is that there are no errors!