Major update to the nanopub API, adding signing in python, removing java dependency

[vemonet]

Hi everyone (and in particular @raar1, @tkuhn and @svenvanderburg), following some issues with how the nanopub library was handling users profile (cf. https://github.com/fair-workflows/nanopub/issues/150) I went through rewriting some parts to make loading profiles and publishing nanopubs more flexible. On the way I also implemented signing and publishing nanopubs directly in python by reusing the trustyuri-python codebase, so it does not require to have java installed

I brought a few changes to how nanopubs are handled, so that will probably be better to publish it as a new major version v2 to make it clear it will most probably break code written previously (at least all the publishing part).

Here is an overview of the changes made:

Documentation

• The documentation website has been updated to use mkdocs
• The doc website is now published to GitHub Pages via the build.yml workflow if the tests pass.
• The code examples according to the changes I made, and added new pages for the new CLI commands, to describe the development setup and auto generated API references.

You can see the updated doc website here: https://vemonet.github.io/nanopub after the merge it will be under https://fair-workflows.github.io/nanopub

Most changes mentioned here have been documented so feel free to check it out and try it!

Profile and config

• Profile private key can be now be loaded from a string instead of only a Path. This can be convenient if someone wants to load keys from anywhere else than a local file (e.g. network). There are still helpers functions to easily load from the profile.yml file.
• The private and public key are now optional, a new keypair will be generated if the private key is missing. And the public key can be generated from the private key automatically if it is missing (orcid_id, name are still mandatory)
• There is now a NanopubConfig object to hold the various arguments used to define the nanopub server where it will be published and which infos to attach to a nanopub.

Signing and Nanopub class

Signing is now done in python, removing the dependency on java!

• I added the whole codebase of https://github.com/trustyuri/trustyuri-python as I needed to make some changes to make it work. Since it was not yet published to pypi, and did not have a documented process to run the tests, I thought it was the best idea at the moment. Considering that this implementation of trusty URI aims to be used with nanopubs it made sense that it is published along it, and it makes it much simpler to bring modifications to the signing process and test them in the future. We could also move the trustyuri folder out of the nanopub package to publish it separately in the future, but it will require additional documentation to explain how to setup and connect the trustyuri and nanopub packages in development

  • Thanks a lot to the authors of trustyuri-python! A lot of work for normalizing and generating the trusty URI was already done, and I could reuse it to sign the nanopubs

• I moved the java_wrapper to the tests so we can still use it to compare signed nanopubs when testing

• The Publication class has been renamed to Nanopub. That makes a lot of "Nanopub" in the code, but it is more consistent and easier to remember

• Signing is now done directly on the Nanopub object (we don't use the client anymore for this): np.sign() and the np object is updated with the signed RDF (cf. the new doc below to see complete examples).

• This is convenient because it allows to define Templates for common nanopublications by inhereting the main Nanopub object, cf. existing template: Nanopub intro, Nanopub Index, Claim, Retract, Update

• The NanopubClient object is now mainly used for search functions (through the grlc API), no changes there

• I also implemented verification of signature and trusty URIs in signed nanopubs. It works with the nanopubs signed by itself and the nanopub-java

CLI

The setup_nanopub_profile CLI was changed to a new CLI with multiple actions:

• np setup prompt the existing CLI workflow to setup your nanopub profile
• np profile check the current user profile used by the nanopub library (~/.nanopub/profile.yml)
• np sign nanopub.trig to sign a nanopub file
• np publish signed.nanopub.trig to publish a nanopub file (signed or not)
• np check signed.nanopub.trig to check if a signed nanopub file is valid

Full documentation here

I also now use Typer instead of click, which is basically a wrapper around click to make use of type annotations, and avoid repeating the types in the click decorator and the function args

Tests and logs

• Changed the uses of print() to use python logging (which enable users of the library to configure the level of logs they want, setting to INFO will show all existing prints)
• A battery of tests was added using the same testsuite as nanopub-java to make sure the nanopubs signed with python are valid and follows the same standards: https://github.com/vemonet/nanopub/blob/sign-in-python/tests/test_testsuite.py#L10
• Most of the existing tests for the client search have been kept
• Updated the build.yml workflow to tests with all python versions from 3.7 to 3.10
• Linting now also check types with mypy (not strict)

You can see a successful test run here: https://github.com/vemonet/nanopub/actions/workflows/build.yml

Project metadata and development setup

• The setup.py + all files for python dev config such as .flake8 have been merged in 1 pyproject.tml file using the hatch build backend (7 files removed from the root folder of the repository! 🍾).
• I updated the setup for development, it does not require any tool that is not built in python: only venv ands pip are required (or you can use hatch to handle install and virtual envs for you). Checkout the development docs page for the complete instructions
• Hatch is a tool for managing python projects, even if it is relatively new, it is published under pypa and already has been adopted by a lot of popular python packages. A major advantage is that it fully complies with the PEP, so the projects metadata are defined using PEP 621 standard
• ⚠️ RDFLib >=6.0.2 is required due to this error triggered when running the SPARQL query to extract nanopub metadata

Feel free to check it out and let me know if you have any comments on the new API, what is not working as expected, and how it could be improved

Otherwise, do you know which libraries are using the nanopub python package? To check if they need to fix their requirements to ask for <2 or update their code with the new nanopub version (and rdflib 6 if not done already). I know only about the fairworkflows lib, I already updated our service to use this new version of nanopub

[tkuhn]

Thanks for this fantastic work, @vemonet!

[raar1]

This is really great work, @vemonet

[raar1]

@vemonet For fixing the failing build docs task, I would try either removing the git pull line that is generating the error or (if it really is needed) perhaps adding something like the following would help:

uses: actions/checkout@v3
with:
  ref: ${{ github.ref_name }}

(I think that would set the current branch to e.g. the PR branch.)

But with the checkout action do we really need the git pull in the first place?

[vemonet]

@raar1 Indeed we can remove the git pull part, I added it at the start because I encountered a conflict, but with the new workflow architecture it is not needed anymore

Thanks for all the feedbacks! I made all the changes in the next commit

[raar1]

Hi @vemonet, is everything essentially ready to be merged now? I see that the docs build is failing but I wonder if that is just because we are merging a fork that does not have the right permissions to make the site on the nanopub repo (or something like that? I'm not sure).

If everything else is ready to go then we can try merging into main, and see if that helps to fix the docs build

[vemonet]

Hi @raar1 , I am quite confident that the docs failing is due to permissions. Which is good! GitHub should not let a pull request trigger an update to the official doc without being merged :)

I documented the process to create a new release in the docs btw: https://vemonet.github.io/nanopub/getting-started/development/#publish-a-new-release and I already upgraded the version to 2.0.0 in the code, so you just need to create the release on GitHub

Feel free to change the docs if you prefer a different process

[raar1]

OK perfect! I will merge now and create a new release :)