Add API function for using cleanly as a library #39

[TG1999]

Reference: https://github.com/nexB/python-inspector/issues/39 Signed-off-by: Tushar Goel tushar.goel.dav@gmail.com

[tdruez]

@TG1999 You resolver_api code is still too correlated with the CLI code.

For example:

json_output = resolver_api(requirement_files=[req_file.name])

ValueError:

No output file specified. Use --json or --json-pdt.

I need a function that, provided a requirements file, returns directly the json as a python object. Also, those errors are irrelevant in the context of using the tool as a library.

[TG1999]

@tdruez Re:

>>> resolver_api(requirement_files=["reqs.txt"])
>>> packages = inspector_output.get("packages")
>>> len(packages)
2

>>> packages[0].get("purl")
'pkg:pypi/toml@0.10.2'
>>> packages[1].get("purl")
'pkg:pypi/toml@0.10.2'

We have one download_url per package, that's why the purl for both of these are same.

[TG1999]

@tdruez Re:

I'm getting an error when using this requirements file as input:
https://github.com/nexB/scancode-toolkit/blob/develop/requirements.txt

>>> inspector_output = resolver_api(requirement_files=['requirements.txt'])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 446, in resolver_api
    resolved_dependencies, purls = resolve(
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 542, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
TypeError: cannot unpack non-iterable NoneType object

I got this error while using this req file :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 461, in resolver_api
    resolved_dependencies, purls = resolve(
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 548, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 598, in get_resolved_dependencies
    raise_error(f"{e!r}")
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 276, in raise_exception
    raise Exception(message)
Exception: Exception('Unable to collect setup.py dependencies securely')

I will suggest you to use this resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

[tdruez]

We have one download_url per package, that's why the purl for both of these are same.

Could you expend a bit your explanation? I do not understand why single line toml==0.10.2 returns 2 duplicated entries in the packages list.

[tdruez]

I will suggest you to use this resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

Could you explain why this issue is occurring? What is the specific part of this requirements file that cause thew issue?

[TG1999]

I will suggest you to use this resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

Could you explain why this issue is occurring? What is the specific part of this requirements file that cause thew issue?

@tdruez rdflib causes this issue, https://github.com/RDFLib/rdflib/blob/main/setup.py

[tdruez]

@TG1999 why "force pushing" into a single commit again? I makes impossible to track the progress made...

[TG1999]

@tdruez we pushed some commits in between that were only for verifying the content of the flaky tests in the CI. And since the verification is now done I squashed the intermediate commits.