search

napari / napari-core

BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Introduce plugin management system #13

Closed kne42 closed 5 years ago

kne42 commented 6 years ago

Description

Introduces a plugin management system (and necessary utility functions for general configuration files).

The specification for this system is as follows:

plugins.yml specification ```json { "$schema": "http://json-schema.org/draft-04/schema#", "fileMatch": [ "plugins.yml" ], "definitions": { "git_url": { "description": "Repository git source URL.", "type": "string", "TODO": "add actual http(s)://{...}.git pattern" } }, "properties": { "plugins": { "description": "List of plugin sources.", "type": "array", "items": { "type": "object", "anyOf": [ { "properties": { "git_source": { "$ref": "#/definitions/git_url" }, "version": { "description": "The git branch/tag/commit to use as a source.", "type": "string" }, "development": { "description": "If enabled, will treat the repository as a normal folder (no version control)." } }, "required": ["git_source"] }, { "properties": { "folder": { "description": "The folder from which the plugin is sourced. Can be used with 'git_source' to handle repository name collisions.", "type": "string" } }, "required": ["folder"] } ] } } }, "required": ["plugins"] } ```
example plugins.yml file ```yaml plugins: - git_source: github.com/jni/foo.git - git_source: github.com/kne42/foo.git folder: bar - folder: baz - git_source: github.com/Napari/hello-world-plugin.git version: 1.0 development: yes ```


napari.yml specification ```json { "$schema": "http://json-schema.org/draft-04/schema#", "fileMatch": [ "napari.yml" ], "definitions": { "git_url": { "description": "Repository git source URL.", "type": "string", "TODO": "add actual http(s)://{...}.git pattern" }, "filepath": { "type": "string", "TODO": "add regex pattern" }, "python_file": { "type": "string", "TODO": "add regex pattern" }, "python_module_name": { "type": "string", "TODO": "add regex pattern" }, "version": { "oneOf": [ { "description": "The exact version.", "type": "string" }, { "description": "The minimum/maximum version.", "type": "object", "properties": { "minimum": { "type": "string" }, "maximum": { "type": "string" }, "min": { "type": "string" }, "max": { "type": "string" } }, "anyOf": [ { "oneOf": [ { "required": ["minimum"] }, { "required": ["min"] } ]}, { "oneOf": [ { "required": ["maximum"] }, { "required": ["max"] } ]}, ] } ] } }, "properties": { "plugin_info": { "description": "Information about the plugin.", "properties": { "author_name": { "description": "The plugin maintainer's name.", "type": "string" }, "author_email": { "description": "The plugin maintainer's email.", "type": "string", "format": "email" }, "short_description": { "description": "A short description of the plugin.", "type": "string" }, "plugin_name": { "description": "The name that the plugin will be available under in napari.plugins.", "$ref": "#/definitions/python_module_name" }, "path": { "description": "Path to the python source file.", "oneOf": [ { "$ref": "#/definitions/python_file" }, { "type": "array", "items": { "$ref": "#/definitions/python_file" } } ] }, "paths": { "description": "Paths to the python source files.", "type": "array", "items": { "$ref": "#/definitions/python_file" } } }, "allOf": [ { "required": ["plugin_name"] }, { "oneOf": [ { "required": ["path"] }, { "required": ["paths"] } ]} ] }, "redirect": { "description": "If this repository is no longer being maintained, please redirect to another one.", "$ref": "#/definitions/git_url" }, "plugin_dependencies": { "description": "A list of depended-on plugins.", "type": "array", "items": { "oneOf": [ { "$ref": "#/definitions/git_url" }, { "type": "object", "properties": { "git_source": { "$ref": "#/definitions/git_url" }, "version": { "$ref": "#/definitions/version" }, "namespace": { "$ref": "#/definitions/python_module_name" } }, "required": ["git_source"] } ] } }, "pip_requirements": { "description": "A requirements.txt file or a list of pip requirements to install.", "oneOf": [ { "description": "Location of the requirements.txt file.", "$ref": "#/definitions/filepath" }, { "description": "pip requirement.", "type": "string" } ] }, "napari_version": { "description": "The exact napari version to specify or the bounds for it.", "$ref": "#/definitions/version" }, "update_script": { "description": "The python script to run upon updating the repository.", "$ref": "#/definitions/python_file" } }, "required": ["plugin_info"] } ```


References

Closes #3

royerloic commented 6 years ago

Really cool Kira. Should I merge already?

kne42 commented 6 years ago

@royerloic Haha, not even close! I like to open up PRs for WIP features so that others can track my progress (and to call dibs on an issue) :P

Besides, I think that we should usually have you and/or @jni review the changes before a merge.

I’ll let you know when it’s ready for its first review!

royerloic commented 6 years ago

Haha, ok cool, sensed that it was just the beginning, but looks already great! I like your attention to details!

kne42 commented 6 years ago

@royerloic @jni I added a proposed plugins specification in the PR description if you want to check that out :)

royerloic commented 6 years ago

Hi Kira,

Very nice, I like that you are considering all sorts of plugin types, from pip-based to git-based…

On 21. Jun 2018, at 07:38, Kira Evans notifications@github.com wrote:

@royerloic https://github.com/royerloic @jni https://github.com/jni I added a proposed plugins specification in the PR description if you want to check that out :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Napari/napari/pull/13#issuecomment-399126745, or mute the thread https://github.com/notifications/unsubscribe-auth/AByMkmN_87Vi2RjSUCZreNJpb4U_Hru2ks5t-6_QgaJpZM4UuDBb.

kne42 commented 6 years ago

An example of the remote plugin updating can be found in this gist.

kevinyamauchi commented 6 years ago

Hey, Kira. This looks awesome! We haven't met yet, but I am an engineer at the Biohub. I was chatting with Loic and I would like to contribute a couple of image loading plugins related to the imaging database at the Biohub and our in situ transcriptomics work.

Would it be possible to chat about the plugin API for IO plugins sometime next week?

Also, a couple of comments on the plugin install:

kne42 commented 6 years ago

@kevinyamauchi

Hey, Kira. This looks awesome! We haven't met yet, but I am an engineer at the Biohub.

Hi Kevin! I'm glad that you're super excited about this project as well 😄

I was chatting with Loic and I would like to contribute a couple of image loading plugins related to the imaging database at the Biohub and our in situ transcriptomics work.

Great! That would be awesome! It would be especially nice if we could get these plugins to load in the proper metadata as well (see #5).

Would it be possible to chat about the plugin API for IO plugins sometime next week?

Definitely! You can find me at Cubicle #164 in the Accelerator on Mondays, Wednesdays, and Fridays.

Also, a couple of comments on the plugin install:

  • Do you think it is necessary to support so many different plugin description/manifest designs (.yaml, init, main.py)? I think if we have a single standard, it's easier to check for compliance and also it's easier to understand new plugins quickly. I suppose the counterpoint is that by being flexible, it may be possible to use some existing plugins/modules without modification.

Actually Loic and I finally had an opportunity to discuss this in person yesterday – I am redesigning the system as we speak! I agree that the current design is far too messy and am working to standardize it by requiring all of the main plugin information to be held within the repository's specification file and simplifying the main index file to comprising of (mostly) a bunch of urls. I'll update the specification in the description and let y'all know of the changes via comment when I'm done!

  • Perhaps a bit early to discuss this, but what do you think about creating base classes that provide the required elements of plugins (perhaps one for each type? io, image transform, etc.) and then plugin contributors create subclasses of the appropriate base class?

Sorry, I don't quite follow. Perhaps you could flush out your idea more in #15?

kne42 commented 6 years ago

Ok, the new specifications have been (roughly) outlined in the PR description.

jni commented 6 years ago

Hi @kevinyamauchi!

I'm excited to see Napari growing so fast already! Clearly a lot of people have been silently wishing for something like this.

Perhaps a bit early to discuss this, but what do you think about creating base classes that provide the required elements of plugins (perhaps one for each type? io, image transform, etc.) and then plugin contributors create subclasses of the appropriate base class?

Over at scikit-image, and to a similar extent at SciPy and NumPy, we favour a functional programming style. I would strongly prefer that plugins were simple type-annotated functions. We can then use the types to automatically generate a GUI dialog.

kevinyamauchi commented 6 years ago

Thanks for the quick responses! @kne42, I'll send you an email about meeting to discuss the plugin API (focusing on image IO).

@jni, that works for me! This project would be super useful for our work and I'd be stoked to contribute!

pep8speaks commented 6 years ago

Hello @kne42! Thanks for updating the PR.

Comment last updated on October 09, 2018 at 00:54 Hours UTC
ctrueden commented 6 years ago

I ask this naively, since I am not a very proficient Python programmer—and apologies if I misunderstand what this PR is trying to accomplish—but would it makes sense to consider building on some general-purpose Python plugin framework like PluginBase?

royerloic commented 6 years ago

I ask this naively, since I am not a very proficient Python programmer—and apologies if I misunderstand what this PR is trying to accomplish—but would it makes sense to consider building on some general-purpose Python plugin framework like PluginBase?

Hi Curtis! Nice to see you here. So, the idea is to have a plugin system that is entirely git based. :-)

kne42 commented 6 years ago

Hi @ctrueden,

Thank you for your input! We have, at this time, not found any suitable existing Python plugin systems to use as a base. The repository that you linked appears to be a convoluted workaround of specifying a __path__ attribute to a module's __init__.py. Additionally, it works by replacing the built-in __import__, which is not only bad practice, but could be implemented much more cleanly with a custom import hook. Regardless, this implementation is not suitable as our import system needs to find and load individual files instead of specifying additional package search locations on the fly.

To answer your other question, the scope of this PR goes far beyond just loading plugins. It uses git to actually install and download plugins (hoping to eventually model it after straight.el) and loads them according to their entry point specifications. It additionally provides a central hub for registering plugins' functions to be populated in the GUI.

ctrueden commented 6 years ago

Thanks for the detailed reply, @kne42. Related question: would this feature be useful as a reusable module, beyond only Napari?

I am also curious about the 5-10 year vision of this project in general, but probably this issue is not the ideal place to discuss that. I commented in #9 accordingly.

kne42 commented 5 years ago

While this isn't perfected, I think it's a good enough start to merge since it is only an "introduction" after all.

royerloic commented 5 years ago

Perfect Kira! step by step 😀

On Oct 11, 2018, at 7:31 AM, Kira Evans notifications@github.com wrote:

While this isn't perfected, I think it's a good enough start to merge since it is only an "introduction" after all.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.