This repository contains the source code for the Biobroker
python package. This package contains
multiple utilities, but the overall philosophy/goal can be described in a phrase:
Given an input file of any type containing metadata about entities, broker it to any archive and return a human-readable result.
I understand that this is a very ambitious project; for starters, the variety of existing archives is way too high. For now, this package is being developed by extending my knowledge on the european genomics/bio archives hosted by EMBL-EBI. It may be that some archives can't just be adapted, but I will be very happy if this library can be extended to just work with enough archives to ensure a biology laboratory can set up an automatic brokering process that works for them.
The infrastructure, in simple terms, is a very modularized submission system. Each of the components should have a generic python object, that collects all the attributes and expected methods for that class. Then, for each archive, a set of subclasses for each (Or some) of those modules must be defined.
Let's take as an example the "authenticator". Very briefly, authenticator is a module that takes in a password and a username and spits out a token. It also defines the requests, to ensure the token is always provided with the headers.
Each subclass of the GenericAuthenticator
defines:
WebinApi
, requests need to have a
specific body)And so on with all the modules.
All these modules interact with each other, but can be used mostly independently. Some modules are dependent
on each other, such as the API module needing the Authenticator in order to send requests. In the end, there
is a module that loads all of them, but should be generic enough that only one needs to be defined: the Wrangler
.
A graphical representation of the overall infrastructure can be seen below:
graph LR;
input_metadata>Input metadata]
field_mapping>Field mapping]
output_file>Output file]
input_processor(Input processor)
metadata_entity(Metadata entity)
broker(Wrangler)
authenticator(Authenticator)
api(API)
output_processor(Output processor)
input_metadata-->|Provide metadata for archive|input_processor
field_mapping-->|Provide transformation rules for metadata fields|input_processor
metadata_entity-->|provide rules to generate and validate valid entities|input_processor
input_processor-->|Process metadata and be able to return a list of entities|api
authenticator-->|Provide token and REQUEST methods|api
api-->|"Provides with endpoints and ready [meta]data to"|broker
output_processor-->|"Provide entity-->record method"|broker
broker-->|Accessions and records saved|output_file
Examples can be found under the examples
folder. WIP
Further documentation, including extended definitions and restrictions/requirements for all classes will be given below; for code-generated documentation, please refer to the docs.
pip3 install biobroker
This library does not support CLI access at the moment (Maybe in the future I could think of a CLI; although I would much rather have a simple GUI). As such, you will have to install and write scripts using the objects
Please see the examples folder.
BROKER_LOG
: Path to store the log for the execution of the library. Defaults to broker_log.txt
if not set.API_ENVIRONMENT
: Environment to be used by the API/Auth service.To generate documentation:
cd biobroker
cd docs
make clean html
This is usually not needed, as there is a GH action set-up to generate the documentation automatically. Use it only if you're not experienced with sphinx/unsure of how changes will look like in the RTD docs.
If you add a new subclass, please add it under the autosummary
directive in the RST file. You can see the examples
in the existing docs/
folder.
If the RST files need to be regenerated, it can be done as so:
sphinx-apidoc -Mf -o docs biobroker
This will regenerate the rst files with module content first! Just as I like it. However, we almost never want to regenerate the RST files - So please don't do this without opening an issue and a related PR.
Regarding documentation:
When adding development notes that are specific to a submodule, please add them under "README.md" within the submodule folder.
When adding user guidelines or user-friendly descriptions for any of the submodules, please fill out the
__init__
file within the submodule folder. User-friendly documentation for the library is generated from
this file (For top/high-level module documentation) and from the code itself (for command/function documentation)
using Sphinx
Regarding new entities:
Generic
of each subclass you want to create. You will need to override some functions and properties!api
object accessing a metadata_entity
method), don't make it privatemetadata_entity
calling a function to process certain data), make
it private.__all__
and as an import