search

proycon / codemetapy

A Python package for generating and working with codemeta
https://codemeta.github.io/
GNU General Public License v3.0
24 stars 5 forks source link

Add ability to filter out extras from setup.py #4

Closed matthewfeickert closed 3 years ago

matthewfeickert commented 3 years ago

Hi. Thanks very much for making codemetapy — it has been quite useful. We're using it for our project pyhf but we have quite a few extras in our setup.py to make it easy for us to setup different environments in CI and elsewhere. At the moment codemetapy tries to include information on all of our extras, which is not necessarily the desired output.

For example, in a new Python virtual environment if you do

(codemeta-example) $ python -m pip install -q --upgrade pip setuptools wheel
(codemeta-example) $ python -m pip install -q codemetapy "pyhf==0.5.2"
(codemeta-example) $ python -m pip list
Package            Version
------------------ -------
attrs              20.2.0
click              7.1.2
CodeMetaPy         0.3.2
importlib-metadata 2.0.0
jsonpatch          1.26
jsonpointer        2.0
jsonschema         3.2.0
nameparser         1.0.6
numpy              1.19.2
pip                20.2.3
pkg-resources      0.0.0
pyhf               0.5.2
pyrsistent         0.17.3
PyYAML             5.3.1
scipy              1.5.2
setuptools         50.3.0
six                1.15.0
tqdm               4.50.0
wheel              0.35.1
zipp               3.3.0
(codemeta-example) $ codemetapy pyhf > codemeta.json
Processing source #1 of 1
Obtaining python package metadata for: pyhf
Found metadata in  /home/feickert/.venvs/codemeta-example/lib/python3.7/site-packages/pyhf-0.5.2.dist-info
WARNING: No translation for distutils key Metadata-Version
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Platform
WARNING: No translation for distutils key Requires-Python
WARNING: No translation for distutils key Description-Content-Type
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra

You end up with a very long codemeta.json that has redundancies in it due to it not parsing ~= correctly and trying to deal with all the extras

(codemeta-example) $ wc -l codemeta.json
1600 codemeta.json

For comparison, the codemeta.json that I ended up creating after removing the entries from "softwareRequirements" that were from extras is

$ wc -l codemeta.json 
127 codemeta.json

If it would be possible to add a --no-extras flag or something along those lines of functionality that would be fantastic. Regardless, thank you for helping make software better!

proycon commented 3 years ago

Thanks for using codemetapy! I see the problem indeed. I'll prepare a fix for it, probably removing all these extras by default or at least ensuring there won't be any duplicates.

proycon commented 3 years ago

I just released v0.3.3 that should hopefully address your issues! Version information is now properly parsed (on a first-come basis, including all the extras by default). And there's a parameter to explicitly skip all the extras. https://github.com/proycon/codemetapy/releases/tag/v0.3.3

Feel free to reopen if there's something not in order.

matthewfeickert commented 3 years ago

@proycon Thanks so much for the super speedy response and patch release! Wow! :D

However, I'm still seeing some extras leak in. pyhf's requirements are just the following

scipy>=1.4.0
click>=6.0
tqdm
jsonschema>=3.2.0
jsonpatch
pyyaml

yet continuing with the example environment above

(codemeta-example) $ python -m pip install -q --upgrade codemetapy
(codemeta-example) $ python -m pip list
Package            Version
------------------ -------
attrs              20.2.0
click              7.1.2
CodeMetaPy         0.3.3
importlib-metadata 2.0.0
jsonpatch          1.26
jsonpointer        2.0
jsonschema         3.2.0
nameparser         1.0.6
numpy              1.19.2
pip                20.2.3
pkg-resources      0.0.0
pyhf               0.5.2
pyrsistent         0.17.3
PyYAML             5.3.1
scipy              1.5.2
setuptools         50.3.0
six                1.15.0
tqdm               4.50.2
wheel              0.35.1
zipp               3.3.0
(codemeta-example) $ codemetapy --no-extras pyhf > codemeta.json
Processing source #1 of 1
Obtaining python package metadata for: pyhf
Found metadata in  /home/feickert/.venvs/codemeta-example/lib/python3.7/site-packages/pyhf-0.5.2.dist-info
WARNING: No translation for distutils key Metadata-Version
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Platform
WARNING: No translation for distutils key Requires-Python
WARNING: No translation for distutils key Description-Content-Type
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
(codemeta-example) $ wc -l codemeta.json
154 codemeta.json

which is longer than it should be as it also includes iminuit, jaxlib, and jax which are extras for pyhf

(codemeta-example) $ sed -n '96,135 p' codemeta.json 
        {
            "@type": "SoftwareApplication",
            "identifier": "iminuit",
            "name": "iminuit",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">=1.4."
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "jaxlib",
            "name": "jaxlib",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">0.1.3"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "jax",
            "name": "jax",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">0.1.5"
        }
    ],
proycon commented 3 years ago

I was a bit too quick I see :) There was indeed a bug. It should be solved now in the new v0.3.4 release I just did. When running on pyhf with --no-extras I get:

$ wc -l codemeta.json
115 codemeta.json

which is a bit less than you initially reported after your manual pruning step. But it includes everything from your setup.cfg and none of the extras. Does it look right to you now?

matthewfeickert commented 3 years ago

Does it look right to you now?

Looking good! Thanks so much! :rocket: 

$ pip list | grep CodeMetaPy
CodeMetaPy         0.3.4
$ codemetapy --no-extras pyhf > codemeta.json
$ sed -n '20,96 p' codemeta.json 
    "softwareRequirements": [
        {
            "@type": "SoftwareApplication",
            "identifier": "scipy",
            "name": "scipy",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">=1.4.0"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "click",
            "name": "click",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">=6.0"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "tqdm",
            "name": "tqdm",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "jsonschema",
            "name": "jsonschema",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3",
            "version": ">=3.2.0"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "jsonpatch",
            "name": "jsonpatch",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3"
        },
        {
            "@type": "SoftwareApplication",
            "identifier": "pyyaml",
            "name": "pyyaml",
            "provider": {
                "@id": "https://pypi.org",
                "@type": "Organization",
                "name": "The Python Package Index",
                "url": "https://pypi.org"
            },
            "runtimePlatform": "Python 3"
        }
    ],