Bitbucket API Update [ENG-303]

[cslzchen]

Ticket

https://openscience.atlassian.net/browse/ENG-303

Purpose

• [x] Bitbucket API 1.0 shutdown -->> Bitbucket API 2.0
• [x] Bitbucket API 2.0 GDPR update
  • Shouldn't affect WB since it gets auth and user info from OSF.
  • [x] Double check Bitbucket add-on in the OSF repo Here is the OSF PR.

Changes

Please refer to the docstr/comments for technical details. We have two JIRA pages [Longze's BB API Doc (Link TBD)]() and Git-Backed-Provider-Interfaces with more info. The unit tests for the provider has been fully refactored.

V2 Repository Endpoints In Scope

• Repo Metadata for Main Branch

  This dedicated endpoint for fetching a repo's main branch provided by V1 is no longer available in V2. Fortunately, we can still obtain it from the repo's own metadata.

• Path

  Bitbucket API 2.0 integrates 4 actions into this endpoint.

  • File download: type is file, no query param format=meta
  • File metadata: type is file, with format=meta as query param
  • Folder content listing: type is folder, no query param format=meta
  • Folder metadata: type is folder, with format=meta as query param

• File History

  • Response pagination with max page size as 100 items

• Using fields= Query Param: The Trick

  • Saves one or two extra request(s) for retrieving details about a commit.
  • Reduces response size by telling BB API to only return attributes that WB needs.

Provider Actions

• File Metadata

  • BB provider used to obtain file metadata from its parent's content list with BB API V1. No with V2, it can use the dedicated file metadata endpoint.
  • However, due to missing timestamp (time last modified) and created_utc (time created) in this new endpoint, the provider must make an extra call to the file history endpoint to get the two timestamps.

• File Download

  • Just endpoint change.

• Folder Metadata and Listing

  • BB API V1 has a dedicated endpoint (unfortunately gone in V2) that returns:
  • the folder's own metadata
  • a list of its subfolders
  • a list of detailed metadata of its child files
  • For the folder's own metadata (node and path), the provider now uses the V2 folder metadata endpoint.
  • For content listing, it uses the new V2 folder content listing endpoint, which provides path for each file/folder and size for each file but misses other details.
  • Note: this new endpoint pages response.
  • For each child file's metadata (other details) such as dates and revision/last-commit, use both the file metadata and history endpoint as mentioned in "File Metadata". However, we decided to removed this for a few reasons:
  • Too many requests / too much performance hit
  • OSF files page never uses dates and revision when listing a folder
  • File metadata no longer uses this folder listing (probably primary reason in previous design)

• File History / Revisions

  • This endpoint uses pagination.

• Validating Path

  • self._parent_dir itself and its usage has been fully refactored due to file/folder metadata and folder listing changes.

V2 GDPR

• WB is not affected by BB V2 GDPR changes since

  • It's OSF's job to pass WB the correct parts to build V2 URL.
  • WB only uses repository endpoints which is not affected by GDPR change, see here.

• Please refer to https://github.com/CenterForOpenScience/osf.io/pull/9024 for potential GDPR changes.

Side effects

• Listing folder contents no longer returns timestamp, created_utc, or revision. This is OK since OSF files page never uses this information at all.
• For a "revived" file, file history / revisions no longer shows the history before it was deleted.

QA Notes

Full regression test on the Bitbucket provider (read-only) is required. It can be tested independently (recommended) with the OSF side update. Here is a list of vital actions that fall in the scope of WB. Some are not applicable on the files widget embedded in the project page. Please use the files page instead.

• Switching branch
• Loading BB storage folder
• Listing directory / subdirectory contents
  • Name and Size should be displayed
  • Version and Modified should be empty
  • Downloads is unrelated to this ticket
• Viewing one file
• Downloading one file
• Viewing a file's revision list
  • Version ID, Date and author should appear
• Viewing one file revision
• Downloading one file revision

Deployment Notes

• [x] Bitbucket add-on in the OSF repo should be updated as well. Deployment is independent.

[coveralls]

Coverage increased (+0.003%) to 91.853% when pulling d4503d1d51b518e692c56b00c49996ef5e05c254 on cslzchen:feature/bitbucket-api-update into 2cc62a1ef2c426238271d5d58a4473acecf0878e on CenterForOpenScience:develop.