search

singer-io / tap-github

A Singer tap for extracting data from the GitHub API
GNU Affero General Public License v3.0
74 stars 88 forks source link

{"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-a-repository"} #112

Open ctcreel opened 3 years ago

ctcreel commented 3 years ago

I attempted to use this tap on a simple repo and received this error -

2021-04-28 22:28:57,287Z   main - INFO Running tap-github version 1.9.2 and target-stitch version 3.2.0
2021-04-28 22:28:57,414Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true )
2021-04-28 22:28:57,416Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true )
2021-04-28 22:28:57,417Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true )
2021-04-28 22:28:57,491Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [18] at offset None
2021-04-28 22:28:57,493Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-github --config /tmp/tap_discover_config.json --discover
2021-04-28 22:28:57,798Z   main - INFO Tap exited normally.
2021-04-28 22:28:57,800Z   main - INFO Saving list of discovered streams
2021-04-28 22:28:57,817Z   main - INFO Saving structure of stream stargazers (tap_stream_id: stargazers)
2021-04-28 22:28:57,837Z   main - INFO Saving structure of stream assignees (tap_stream_id: assignees)
2021-04-28 22:28:57,849Z   main - INFO Saving structure of stream events (tap_stream_id: events)
2021-04-28 22:28:57,868Z   main - INFO Saving structure of stream reviews (tap_stream_id: reviews)
2021-04-28 22:28:57,885Z   main - INFO Saving structure of stream commits (tap_stream_id: commits)
2021-04-28 22:28:57,903Z   main - INFO Saving structure of stream pull_request_reviews (tap_stream_id: pull_request_reviews)
2021-04-28 22:28:57,922Z   main - INFO Saving structure of stream teams (tap_stream_id: teams)
2021-04-28 22:28:57,940Z   main - INFO Saving structure of stream team_memberships (tap_stream_id: team_memberships)
2021-04-28 22:28:57,952Z   main - INFO Saving structure of stream project_cards (tap_stream_id: project_cards)
2021-04-28 22:28:57,969Z   main - INFO Saving structure of stream issue_events (tap_stream_id: issue_events)
2021-04-28 22:28:57,997Z   main - INFO Saving structure of stream issue_labels (tap_stream_id: issue_labels)
2021-04-28 22:28:58,017Z   main - INFO Saving structure of stream issues (tap_stream_id: issues)
2021-04-28 22:28:58,042Z   main - INFO Saving structure of stream review_comments (tap_stream_id: review_comments)
2021-04-28 22:28:58,067Z   main - INFO Saving structure of stream issue_milestones (tap_stream_id: issue_milestones)
2021-04-28 22:28:58,085Z   main - INFO Saving structure of stream project_columns (tap_stream_id: project_columns)
2021-04-28 22:28:58,101Z   main - INFO Saving structure of stream releases (tap_stream_id: releases)
2021-04-28 22:28:58,123Z   main - INFO Saving structure of stream team_members (tap_stream_id: team_members)
2021-04-28 22:28:58,145Z   main - INFO Saving structure of stream commit_comments (tap_stream_id: commit_comments)
2021-04-28 22:28:58,163Z   main - INFO Saving structure of stream collaborators (tap_stream_id: collaborators)
2021-04-28 22:28:58,178Z   main - INFO Saving structure of stream pr_commits (tap_stream_id: pr_commits)
2021-04-28 22:28:58,198Z   main - INFO Saving structure of stream comments (tap_stream_id: comments)
2021-04-28 22:28:58,216Z   main - INFO Saving structure of stream pull_requests (tap_stream_id: pull_requests)
2021-04-28 22:28:58,239Z   main - INFO Saving structure of stream projects (tap_stream_id: projects)
2021-04-28 22:28:58,260Z   main - INFO Writing catalog to file
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream issues: labels_url, id, events_url, repository_url, pull_request, comments_url, title, node_id, author_association, number, body, _sdc_repository, html_url, closed_at, updated_at, created_at, locked, assignee, comments, user, url, labels, state
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream collaborators: _sdc_repository, id, url, login, type
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream commit_comments: id, html_url, line, commit_id, created_at, node_id, body, updated_at, path, position, user, url
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream teams: id, members_url, html_url, parent, privacy, repositories_url, node_id, slug, _sdc_repository, name, permission, url, description
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream assignees: _sdc_repository, id, url, login, type
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream issue_milestones: labels_url, id, closed_issues, html_url, closed_at, updated_at, created_at, node_id, number, due_on, open_issues, _sdc_repository, url, creator, title, state, description
2021-04-28 22:28:58,306Z   main - INFO Selected fields for stream comments: id, html_url, author_association, created_at, node_id, updated_at, issue_url, home_url, _sdc_repository, body, user, url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream events: id, actor, push_id, head, payload, size, repo, public, created_at, _sdc_repository, distinct_size, org, ref, type
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream issue_events: label, id, actor, review_requester, node_id, performed_via_github_app, commit_id, rename, event, requested_reviewer, assignee, commit_url, issue, created_at, assigner, _sdc_repository, url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream pull_requests: id, merged_at, base, closed_at, title, created_at, updated_at, number, _sdc_repository, body, user, url, labels, state
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream commits: sha, id, parents, commit, pr_number, files, _sdc_repository, pr_id, url, comments_url, html_url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream releases: published_at, draft, id, html_url, created_at, prerelease, author, body, tag_name, target_commitish, name, _sdc_repository, url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream issue_labels: id, default, color, node_id, name, _sdc_repository, url, description
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream pull_request_reviews: submitted_at, id, html_url, pull_request_url, body, user, commit_id, _links, state, node_id
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream review_comments: in_reply_to_id, id, review_comment_url, base, pull_request_review_id, assignees, patch_url, merge_commit_sha, comments_url, milestone, node_id, body, updated_at, path, requested_reviewers, pull_request_url, position, _sdc_repository, locked, original_commit_id, review_comments_url, statuses_url, diff_hunk, requested_teams, html_url, head, assignee, diff_url, commit_id, author_association, created_at, original_position, issue_url, user, labels, commits_url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream team_members: id, organizations_url, following_url, received_events_url, events_url, subscriptions_url, gists_url, followers_url, node_id, starred_url, site_admin, repos_url, gravatar_id, login, html_url, _sdc_repository, url, type, avatar_url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream pr_commits: sha, id, parents, commit, pr_number, files, _sdc_repository, pr_id, url, comments_url, html_url
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream reviews: submitted_at, id, pull_request_url, html_url, user, body, _sdc_repository, commit_id, state
2021-04-28 22:28:58,307Z   main - INFO Selected fields for stream team_memberships: _sdc_repository, url, state, role
2021-04-28 22:28:58,312Z   main - INFO Current state: {}
2021-04-28 22:28:58,312Z   main - INFO Starting tap: tap-env/bin/tap-github --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2021-04-28 22:28:58,321Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2021-04-28 22:28:58,556Z    tap - INFO Starting sync of repository: medicaid-fl-baha-num46664
2021-04-28 22:28:58,601Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04317307472229004, tags={'status': 'failed', 'endpoint': 'assignees'})
2021-04-28 22:28:58,773Z target - INFO Using batch_size_prefernces of {'full_table_streams': [], 'user_batch_size_preference': None, 'batch_size_preference': None}
2021-04-28 22:28:58,773Z target - INFO Requests complete, stopping loop
2021-04-28 22:28:59,426Z    tap - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [1] at offset None
2021-04-28 22:28:59,426Z    tap - INFO replicated 0 records from "assignees" endpoint
2021-04-28 22:28:59,426Z    tap - CRITICAL {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-a-repository"}
2021-04-28 22:28:59,426Z    tap - Traceback (most recent call last):
2021-04-28 22:28:59,426Z    tap -   File "tap-env/bin/tap-github", line 33, in <module>
2021-04-28 22:28:59,426Z    tap -     sys.exit(load_entry_point('tap-github==1.9.2', 'console_scripts', 'tap-github')())
2021-04-28 22:28:59,426Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 225, in wrapped
2021-04-28 22:28:59,427Z    tap -     return fnc(*args, **kwargs)
2021-04-28 22:28:59,427Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 983, in main
2021-04-28 22:28:59,427Z    tap -     do_sync(args.config, args.state, catalog)
2021-04-28 22:28:59,427Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 956, in do_sync
2021-04-28 22:28:59,427Z    tap -     state = sync_func(stream_schema, repo, state, mdata)
2021-04-28 22:28:59,427Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 726, in get_all_assignees
2021-04-28 22:28:59,427Z    tap -     'https://api.github.com/repos/{}/assignees'.format(repo_path)
2021-04-28 22:28:59,427Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 116, in authed_get_all_pages
2021-04-28 22:28:59,427Z    tap -     r = authed_get(source, url, headers)
2021-04-28 22:28:59,427Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 109, in authed_get
2021-04-28 22:28:59,427Z    tap -     raise NotFoundException(resp.text)
2021-04-28 22:28:59,427Z    tap - tap_github.NotFoundException: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-a-repository"}
2021-04-28 22:28:59,427Z   main - INFO Target exited normally with status 0
2021-04-28 22:29:00,428Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [10] at offset None
2021-04-28 22:29:00,429Z   main - INFO No tunnel subprocess to tear down
2021-04-28 22:29:00,429Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "{"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-a-repository"}". Target succeeded.
ctcreel commented 3 years ago

Note that I do not think this is a token issue. I'm using the same token that I am also using in a Python application to pull this data. That python application works just fine.