Fix so this runs in the meltano demo project

aaronsteers commented 2 years ago

This addresses the issues causing the demo project to fail.

What I found was that we were using project_path and group_path for the endpoints. The docs (at least for a few of the endpoints) say you can use project_id or an escaped version of the project name/path - with some endpoints seeming to to inconsistently be bothers if / in a project path like meltano/meltano is escaped.

Some endpoints that were failing for me before would succeed if I swapped out the project or group path for the ID.

Related to: https://gitlab.com/meltano/demo-project/-/merge_requests/17 which is being worked on in parallel

aaronsteers commented 2 years ago

Documenting here: The legacy tap was failing silently on project labels:

2m2022-05-09T22:49:20.132511Z[0m [[32m[1minfo     [0m] [1mINFO GET https://gitlab.com/api/v4/projects/7603319/labels[0m [36mcmd_type[0m=[35mextractor[0m [36mjob_id[0m=[35mgitlab-to-jsonl[0m [36mname[0m=[35mtap-gitlab[0m [36mrun_id[0m=[35mf29951e5-2d1c-4add-a769-ddbde464086e[0m [36mstdio[0m=[35mstderr[0m
[2m2022-05-09T22:49:20.133420Z[0m [[32m[1minfo     [0m] [1mINFO Skipping request to https://gitlab.com/api/v4/projects/7603319/labels[0m [36mcmd_type[0m=[35mextractor[0m [36mjob_id[0m=[35mgitlab-to-jsonl[0m [36mname[0m=[35mtap-gitlab[0m [36mrun_id[0m=[35mf29951e5-2d1c-4add-a769-ddbde464086e[0m [36mstdio[0m=[35mstderr[0m
[2m2022-05-09T22:49:20.134089Z[0m [[32m[1minfo     [0m] [1mINFO Reason: 401 - b'{"message":"401 Unauthorized"}'[0m [36mcmd_type[0m=[35mextractor[0m [36mjob_id[0m=[35mgitlab-to-jsonl[0m [36mname[0m=[35mtap-gitlab[0m [36mrun_id[0m=[35mf29951e5-2d1c-4add-a769-ddbde464086e[0m [36mstdio[0m=[35mstderr[0m

I'm going to remove labels from the demo project's selected streams until we have positive confirmation that this endpoint is working with those creds.

UPDATE: This same error appears in legacy logs for these streams:

project_lables
project_members
project_milestones
project_variables
vulnerabilities
users

laurentS commented 2 years ago

Tangentially related, we just updated tap-github to use project IDs instead of paths for stability reasons (we were getting problems with repos being renamed). I think what you're doing in this PR is going in the right direction :)

sonarcloud[bot] commented 2 years ago

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

MeltanoLabs / tap-gitlab

Fix so this runs in the meltano demo project #77