search

MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
11 stars 29 forks source link

Readme can have different formats #85

Open ericboucher opened 2 years ago

ericboucher commented 2 years ago

The Readme stream assumes that we are fetching an .md file but it is not always the case.

Eg. README.rst for flake8-docstrings which leads to a 404 file not found in the tap.

{'type': 'timer', 'metric': 'http_request_duration', 'value': 0.164145, 'tags': {'endpoint': '/projects/{project_id}/repository/files/README%2Emd/raw', 'http_status_code': 404, 'status': 'failed', 'url': '/api/v4/projects/140322/repository/files/README.md/raw', 'context': {'project_id': 140322, 'project_path': 'pycqa/flake8-docstrings'}}}

We could:

  1. use the readme_url available in the project object
  2. try different exetnsion, probably .md and .rst would cover 99.9% of cases

But at the very least we should add error handling around missing readmes :)