search

singer-io / tap-codat

A Singer.io tap to extract data from the Codat API
GNU Affero General Public License v3.0
2 stars 7 forks source link

Linking error code 200 on tap to "companies" #13

Open Ivan-Rho opened 3 years ago

Ivan-Rho commented 3 years ago

I tried running the connection for the first time and received an error code 200 on the tap - INFO HTTP request to the "companies" endpoint. Does anyone know what might have caused that? Error logs below.

2021-08-02 14:57:05,219Z main - INFO Running tap-codat version 0.5.0 and target-stitch version 3.2.0 2021-08-02 14:57:06,357Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true ) 2021-08-02 14:57:06,360Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true ) 2021-08-02 14:57:06,360Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true ) 2021-08-02 14:57:07,397Z main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [24] at offset None 2021-08-02 14:57:07,399Z main - INFO Starting tap to discover schemas: tap-env/bin/tap-codat --config /tmp/tap_discover_config.json --discover 2021-08-02 14:57:09,384Z tap - INFO HTTP request to "companies" endpoint took 1.718s, returned status code 200 2021-08-02 14:57:09,431Z main - INFO Tap exited normally. 2021-08-02 14:57:09,432Z main - INFO Saving list of discovered streams 2021-08-02 14:57:09,505Z main - INFO Saving structure of stream companies (tap_stream_id: companies) 2021-08-02 14:57:09,617Z main - INFO Saving structure of stream accounts (tap_stream_id: accounts) 2021-08-02 14:57:09,711Z main - INFO Saving structure of stream bank_accounts (tap_stream_id: bank_accounts) 2021-08-02 14:57:09,738Z main - INFO Saving structure of stream bank_account_transactions (tap_stream_id: bank_account_transactions) 2021-08-02 14:57:09,752Z main - INFO Saving structure of stream bank_statements (tap_stream_id: bank_statements) 2021-08-02 14:57:09,779Z main - INFO Saving structure of stream bank_statement_lines (tap_stream_id: bank_statement_lines) 2021-08-02 14:57:09,801Z main - INFO Saving structure of stream bills (tap_stream_id: bills) 2021-08-02 14:57:09,820Z main - INFO Saving structure of stream company_info (tap_stream_id: company_info) 2021-08-02 14:57:09,900Z main - INFO Saving structure of stream credit_notes (tap_stream_id: credit_notes) 2021-08-02 14:57:09,923Z main - INFO Saving structure of stream customers (tap_stream_id: customers) 2021-08-02 14:57:09,954Z main - INFO Saving structure of stream payments (tap_stream_id: payments) 2021-08-02 14:57:09,989Z main - INFO Saving structure of stream suppliers (tap_stream_id: suppliers) 2021-08-02 14:57:10,013Z main - INFO Saving structure of stream connections (tap_stream_id: connections) 2021-08-02 14:57:10,086Z main - INFO Saving structure of stream bill_payments (tap_stream_id: bill_payments) 2021-08-02 14:57:10,188Z main - INFO Saving structure of stream invoices (tap_stream_id: invoices) 2021-08-02 14:57:10,210Z main - INFO Saving structure of stream journal_entries (tap_stream_id: journal_entries) 2021-08-02 14:57:10,234Z main - INFO Saving structure of stream items (tap_stream_id: items) 2021-08-02 14:57:10,260Z main - INFO Saving structure of stream tax_rates (tap_stream_id: tax_rates) 2021-08-02 14:57:10,277Z main - INFO Saving structure of stream events (tap_stream_id: events) 2021-08-02 14:57:10,289Z main - INFO Saving structure of stream balance_sheets (tap_stream_id: balance_sheets) 2021-08-02 14:57:10,312Z main - INFO Saving structure of stream profit_and_loss (tap_stream_id: profit_and_loss) 2021-08-02 14:57:10,337Z main - INFO Starting tap: tap-env/bin/tap-codat --config /tmp/tap_config.json 2021-08-02 14:57:12,181Z tap - INFO HTTP request to "companies" endpoint took 1.573s, returned status code 200 2021-08-02 14:57:13,283Z tap - INFO HTTP request to "companies" endpoint took 1.097s, returned status code 200 2021-08-02 14:57:13,284Z tap - CRITICAL 'companies' 2021-08-02 14:57:13,284Z tap - Traceback (most recent call last): 2021-08-02 14:57:13,284Z tap - File "tap-env/bin/tap-codat", line 33, in 2021-08-02 14:57:13,285Z tap - sys.exit(load_entry_point('tap-codat==0.5.0', 'console_scripts', 'tap-codat')()) 2021-08-02 14:57:13,285Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_codat/init.py", line 106, in main 2021-08-02 14:57:13,285Z tap - main_impl() 2021-08-02 14:57:13,285Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_codat/init.py", line 101, in main_impl 2021-08-02 14:57:13,285Z tap - sync(ctx) 2021-08-02 14:57:13,285Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tapcodat/init.py", line 77, in sync 2021-08-02 14:57:13,285Z tap - streams.companies.fetch_into_cache(ctx) 2021-08-02 14:57:13,285Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_codat/streams.py", line 141, in fetch_into_cache 2021-08-02 14:57:13,285Z tap - ctx.cache["companies"] = self.transform_dts(ctx, resp["companies"]) 2021-08-02 14:57:13,285Z tap - KeyError: 'companies' 2021-08-02 14:57:13,315Z main - INFO Connection check failed 2021-08-02 14:57:14,452Z main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [21] at offset None 2021-08-02 14:57:14,455Z main - INFO No tunnel subprocess to tear down 2021-08-02 14:57:14,455Z main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "'companies'".

mzsleo85 commented 3 years ago

Same issue for me if any one can help

mattarderne commented 3 years ago

Also getting this issue

mattarderne commented 3 years ago

@mzsleo85 @Ivan-Rho if you are paid customers of Codat and Stitch please request them both to look into this, I think possibly minor fix

aevv commented 3 years ago

this is likely caused by removal of un-paginated endpoints Codat side, and this project not handling the newer response bodies. See Codat swagger for Companies for an example of paginated response bodies, which were introduced in the last few months

it looks like companies (and some other datatypes) are not using pagination which appears to be supported for some datatypes that did support it in the past, so changing those to go via the Pagination class might fix this

visch commented 2 years ago

Fixed this error here https://github.com/singer-io/tap-codat/pull/14

Note that there were still stream failures, had to ignore the streams listed below to get the tap to run without errors. Note that I noticed some other streams didn't fail the entire tap but did report 40x errors, not certain how this tap works and if that's expected or not. 

    select:
    - '!bank_accounts.*'
    - '!bank_statements.*'
    - '!connections.*'
    - '!journal_entries.*'
    - '!items.*'
    - '!tags.*'
    - '!tax_rates.*'
    - '*.*'