Type of API to use

[civier]

Hi @yjmantilla

Another important issue that I raised before is the type of API we're going to provide (python, CLI, REST, etc.). Ideally, we should go to something more robust like a python API (with named parameters, data types, etc.), but on the other hand, I wouldn't like to limit the GUIs to python-based ones.

@stebo85 @aswinnarayanan @civier @DavidjWhite33 @TomEmotion Do you have any suggestions for a robust API type that is not limited to a specific programming language? Maybe something that uses XML, YAML, etc. for specifying the parameters, and has implementations for a wide-range of programming languages? It's really not my expertise, so I'm calling for help.

@yjmantilla I would also consult with the Brainlife/Easybidz on what type of API they would like to see? Similarly, I will consult with Marcel from BIDScoin about something that he might like to adopt for his project as well at one point or another.

Best, Oren

[yjmantilla]

@civier

Another important issue that I raised before is the type of API we're going to provide (python, CLI, REST, etc.). Ideally, we should go to something more robust like a python API (with named parameters, data types, etc.), but on the other hand, I wouldn't like to limit the GUIs to python-based ones.

Thats a good point.

As of now the CLI kind of serves as a way for an end-user to use sovabids in a stand-alone way. So I haven't really though of he CLI being the main API which is the python one.

I don't know much about REST APIs but from what I read when researching bids-toolbox (see #2 ) it is my understanding that one can do a REST API using python, thus it is actually the same as the python API (? I'm not sure).

If anybody knows about this , the clarification would be appreciated.

I would also consult with the Brainlife/Easybidz on what type of API they would like to see?

Yeap, that was what I kind of wanted to do from the beginning with them. They were really busy though so they couldn't at that time say how such API should be. I think it may be possible to schedule a meeting with them showing our current API and take suggestions from them.

Similarly, I will consult with Marcel from BIDScoin about something that he might like to adopt for his project as well at one point or another.

What Marcel has is pretty well-formed by now. Through the plugin abstraction we are able to interface with bidscoin. Here is comparison of the bidscoin-plugin-api to our api:

bidscoin	sovabids
is_sourcefile(file: Path) -> str	equivalent to filtering by extension, we don’t have an api explicetly for this though
get_attribute(dataformat: str, sourcefile: Path, attribute: str, options: dict) -> str	equivalent to the "preview" option of apply_rules_to_single_file
bidsmapper_plugin(session: Path, bidsmap_new: dict, bidsmap_old: dict, template: dict, store: dict) -> None	equivalent to apply_rules (which applies the rules to each file involved and outputs the mappings for each)
bidscoiner_plugin(session: Path, bidsmap: dict, bidsfolder: Path) -> None	equivalent to "convert"

So as of now we don't really have much to propose. The changes I would do to bidscoin are:

• not having a constrain for the input directory structure
• having a rescan button which is something that was discussed here https://github.com/Donders-Institute/bidscoin/issues/87
• having a way to preview bidsfiles, right now one may do that with attributes in a hacky and ugly way as shown here:

[civier]

hello @yjmantilla Still no time to go over the details, but I think I found a really good example online for an API for a conversion tool: https://cloudconvert.com/api/v2/quickstart They have implementations for different languages, but I believe the underlying protocol is REST, with JSON being the data format for input and output parameters of the available functions. Maybe you can take a look before we get the feedback from the other mentors?

[aswinnarayanan]

I don't know much about REST APIs but from what I read when researching bids-toolbox (see #2 ) it is my understanding that one can do a REST API using python, thus it is actually the same as the python API (? I'm not sure).

If anybody knows about this , the clarification would be appreciated.

@yjmantilla Yes, REST api is a style of api; it doesn't specify any language, you can write one in pretty much anything.

API is just a generic term for any interface that communicates with a backend via specific endpoints. Your sovabids api design and the bidscoin api are python libraries. which would be the most straight-forward type of api for this case. But a python library would require the client applications (CLI or GUI) be in python, or be able to talk to python.

Do you have any suggestions for a robust API type that is not limited to a specific programming language? Maybe something that uses XML, YAML, etc. for specifying the parameters, and has implementations for a wide-range of programming languages? It's really not my expertise, so I'm calling for help.

Web apis usually communicate via http protocol so should be viable.

REST is typically used for web and CRUD operations, it could be tricky to map to sovabids. The cloudconvert example could be a good test, since its a converter like sovabids. If the cloudconvert endpoints easily map out to those of sovabids, it might be a good starting point.

If you wanted a alternative to REST, maybe have a look at RPC (or specifically JSON-RPC) which is more object and action driven, like sovabids. This article outlines some pros/cons between REST and RPC for certain applications.

If you're not committed to full-blown web api development at this stage, I could suggest proceeding with development on the python library. And later you could build a web api wrapper around the python api.

[yjmantilla]

@aswinnarayanan @civier thank you for both of your inputs. I think in conclusion one of the goals of this issue will be this one:

The cloudconvert example could be a good test, since its a converter like sovabids. If the cloudconvert endpoints easily map out to those of sovabids, it might be a good starting point.

Regarding:

If you're not committed to full-blown web api development at this stage, I could suggest proceeding with development on the python library. And later you could build a web api wrapper around the python api.

Gonna delve into the cloud convert example and from that I will decide what to do.

[yjmantilla]

If you wanted a alternative to REST, maybe have a look at RPC (or specifically JSON-RPC) which is more object and action driven, like sovabids. This article outlines some pros/cons between REST and RPC for certain applications.

Took a look at the article, certainly seems like it may be a better approach for the sovabids case. After delving into the cloud convert api and mirroring it for sovabids, im gonna draft how would it be in RPC and decide from that

[civier]

@aswinnarayanan Thank you very much for the input on the RPC protocols. This is enlightening.

I was looking a bit on the pages of JSON-RPC and didn't see too much activity lately (https://www.jsonrpc.org/; https://json-rpc.info/). Did this protocol actually catch up, or is it a very marginal thing? I'm afraid that we choose something that is good, but unfamiliar to people (like many things in software, often the things that succeed are not the best or optimal ones, but rather, those that were adopted by large companies/bodies). I also looked at XML-RPC that had a new implementation for javascript in 2019 (http://xmlrpc.com/), but even there, there is not much activity in the github this year.

Shouldn't we better go with REST that is very standard, and just use a very good Python wrapper that will make things almost transparent? (as the wrapper provided by cloudconvert) In any case, for the GSoC I don't think we should deal with anything outside python, so for me the dilemma is between two options: 1) using a plain python API, or 2) using a python API that perfectly wraps an underlying cross-language protocol like REST I would go for the second option if the overhead is not high, and choose REST simply because it's so ubiquitous. I guess we can always switch to a different underlying protocol down the road (if we really decide that REST is not optimal), because the python wrapper for REST is likely to be adopted to other underlying protocols (and not the other way around).

Any thoughts?

@yjmantilla Can you check what python wrappers are available for REST, and if any of them support multiple underlying protocols? Maybe that would be the best option.

[aswinnarayanan]

Hi @civier

Those jsonrpc and xmlrpc links look like their original specifications. A more modern revision would be the OpenRPC spec, compared to OpenAPI, which is the most popular REST api spec. Both have had their latest revision in early 2020, so they're pretty comparable.

I understand that REST api is ubiquitous, but most web services are CRUD operations. sovabids has no database, and is very action driven (atleast right now).

For instance, for sovabids to run a map on a single file right now, it would call the following function: apply_rules_to_single_file(file1)

The respective endpoints would look like thing like this RPC : POST /applyrules {"file": "file1"} REST: POST /file/1 {"rules": "apply"}

And for rescan RPC : POST /rescan {"source", "dir1} REST: POST /sourcedir/1 {"rescan": "true"}

Either way, the choice of api spec is largely work for the sovabids backend developer. For the user or client dev, they're just going to look up the api documenation, and follow whichever syntax is provided., it's not going to affect them except for readability. Either spec can use http transport. Happy for yjmantilla to proceed with whichever way makes sense for the project.

@yjmantilla, if you're looking for python frameworks to develop the web API, fastapi could be a good option. There's plenty of tutorials online for building REST fastapi. And there is one here for json-rpc.

[yjmantilla]

So I got a into REST and RPC (mainly through https://www.youtube.com/watch?v=hkXzsB8D_mo and the article @aswinnarayanan mentioned). Now that I kind of understood the semantics I was able to interpret the cloudconvert API, indeed for me it seems to be RPC instead of REST; which makes sense since it is action-oriented. Here is their endpoint for conversion as an example:

https://cloudconvert.com/api/v2/convert#convert-tasks

As of now I'm playing a bit with fastapi and the jsonrpc version of it. The idea is to replicate the API shown in #25

[yjmantilla]

Update,

I managed to do an early version of the RPC API , along with some tests.

The API is in https://github.com/yjmantilla/sovabids/blob/main/sovabids/sovarpc.py

The tests are in https://github.com/yjmantilla/sovabids/blob/main/tests/test_bids.py , whenever mode=='rpc' is used.

Does it work? Yes, at least locally. I was worried that some trouble with the filesystem would arise. Nevertheless the files are written and read accordingly, I guess this is the case because Im running this on a local server.

Example with load_rules action

Request

Response

The file system and an online server

For an online server I would guess the user uploads the root input folder (or the files themselves) and then that files can be read by the server, then the server writes the output to its own filesystem and it is returned to the user as downloads.

@civier @aswinnarayanan @stebo85 Let me know if you have some feedback on this, specially on how would this work online (given that a server does not have access to the local file system)

[stebo85]

Looks good to me :) I agree, on a remote server you wouldn't be able to interact with anything on the client side, so you would have to download the converted files.

[yjmantilla]

Closing this for now since I decided to go with the RPC way.