WaterButler is a Python web application for interacting with various file storage services via a single RESTful API, developed at Center for Open Science.
Apache License 2.0
62
stars
76
forks
source link
[ENG-333] [OATHPIT] Throw GitLab into the Oath Pit #393
Enable and update the GitLab provider for aiohttp3.
Changes
Fixed download and its tests
This provider has been using a complicated hack for download due to GL-Issue#31470 which breaks the dedicated endpoint for direct raw file download. Fortunately, the bug was fixed in GL API 10.0 so that download has been re-written to use the preferred and intuitive approach.
Another reason for the re-writing is that the work-around would not work with aiohttp3 even if we kept it for backward compatibility. It needs to modify response headers directly within the object before passing it to the response stream reader. However, aiohttp3 makes it impossible: the headers is defined as a @reify property. Neither does using _headers instead of headers work due to the cache effect of @reify property.
Default and range download tests are rewritten and work-around tests has been removed.
Removed another work-around for GitLab file API's ruby hash bug.
Fixed and update fixture usage in tests
Added missing dependencies
I ran into a weird issue where flake8 fails with library import failure. I tried the following three but none of them solves the problem: 1) install and reinstall flake8 inside the container, 2) re-install requirements with docker-compose and 3) invoke install -d inside container. However, removing and re-installing some of it's dependencies works. This may be just a glitch but I decided to add the missing dependencies with the default (most recent) compatible version for flake8==3.5.0.
Side effects
In the following commit message, the fix for download may break GL instances that uses an API older than 10.0. The ruby hash bug has been removed as well given it is even older (fixed in 9.5).
update comments about outstanding GL bugs
Some have been fixed in subsequent releases. Update the comments to reflect this, but do not yet remove the workarounds. Need to evaluate the popularity of old versions in the wild before removing workarounds.
QA Notes
Dev Tests
As usual, no comments indicates a PASS.
Getting metadata for a file / folder: tested along folder listing and file rendering
Downloading
Uploading: N / A
DAZ
Deleting: N / A
Folder
Creation: N / A
Upload: N / A
Deletion: N / A
Rename files and folders: N / A
Verifying non-root folder access for id-based folders: N / A
Intra move / copy: N / A
Inter move / copy (light testing only)
One and multiple files
One and multiple folders
From GitLab to OSFStorage only
Comments persist with moves (light testing only)
If enabled, test revisions: enabled and tested
Project root is storage root vs. a subfolder: N / A
Ticket
https://openscience.atlassian.net/browse/ENG-333
Purpose
Enable and update the GitLab provider for
aiohttp3.Changes
Fixed download and its tests
aiohttp3even if we kept it for backward compatibility. It needs to modify response headers directly within the object before passing it to the response stream reader. However,aiohttp3makes it impossible: theheadersis defined as a@reifyproperty. Neither does using_headersinstead ofheaderswork due to the cache effect of@reifyproperty.Removed another work-around for GitLab file API's ruby hash bug.
Fixed and update fixture usage in tests
Added missing dependencies
docker-composeand 3)invoke install -dinside container. However, removing and re-installing some of it's dependencies works. This may be just a glitch but I decided to add the missing dependencies with the default (most recent) compatible version forflake8==3.5.0.Side effects
In the following commit message, the fix for download may break GL instances that uses an API older than 10.0. The ruby hash bug has been removed as well given it is even older (fixed in 9.5).
QA Notes
Dev Tests
As usual, no comments indicates a PASS.
Getting metadata for a file / folder: tested along folder listing and file rendering
Downloading
Uploading: N / A
DAZ
Deleting: N / A
Folder
Rename files and folders: N / A
Verifying non-root folder access for id-based folders: N / A
Intra move / copy: N / A
Inter move / copy (light testing only)
Comments persist with moves (light testing only)
If enabled, test revisions: enabled and tested
Project root is storage root vs. a subfolder: N / A
Updating a file: N / A
Deployment Notes
No