datalad / datalad-crawler

DataLad extension for tracking web resources as datasets

http://datalad.org

Other

5 stars 16 forks source link

Add LORIS pipelines to crawl LORIS' projects and candidates endpoints #103

Closed cmadjar closed 2 years ago

cmadjar commented 3 years ago

Description

This is a refactoring of the scripts that were present in #102 and #13. One of the pipeline present in #67 has also been added to the PR but not changed.

It contains two files:

loris-projects-api.py: to query the different LORIS endpoints specific to projects (a.k.a. under /projects/) - still needs work
loris-candidate-api.py: to query the LORIS endpoints specific to a candidate (a.k.a. under /candidates/) - for now this is an exact copy of what was in #67 that was submitted by @mathdugre a while back

codecov[bot] commented 3 years ago

Codecov Report

Merging #103 (b313073) into master (7905869) will decrease coverage by 2.20%. The diff coverage is 22.22%.

@@            Coverage Diff             @@
##           master     #103      +/-   ##
==========================================
- Coverage   81.33%   79.12%   -2.21%     
==========================================
  Files          57       59       +2     
  Lines        4650     4829     +179     
==========================================
+ Hits         3782     3821      +39     
- Misses        868     1008     +140

Impacted Files	Coverage Δ
datalad_crawler/pipelines/loris-projects-api.py	`15.10% <15.10%> (ø)`
datalad_crawler/pipelines/loris-candidate-api.py	`45.00% <45.00%> (ø)`
datalad_crawler/version.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 03710b8...b313073. Read the comment docs.

cmadjar commented 3 years ago

@yarikoptic I think you can close #13 since the loris-project-api.py script was based out of it and has been refactored multiple times since the PR was sent. The last refactoring was to try to render it more flexible so that we do not have to create a new pipeline every time we need to query a different projects LORIS endpoint, which is what we've been doing so far...

67 can probably be closed as well if @mathdugre is happy to have his script ported in this PR. As you guys wish.

If you have initial feedbacks, don't hesitate. The loris-project-api.py pipeline is still a work in progress.

yarikoptic commented 2 years ago

Per @mih insistence of seeking for ultimate purity in all of our lives, I am closing this PR due to being inactive for considerable amount of time. But "closed" is not "gone forever". If there is an interest to resurrect this, or alternative/complimentary #13 -- just let us know.