Ingest failing: year is out of range

[nmccormick-bombas]

Any ideas on how to fix this issue? It's the first time I've seen it.

[luandy64]

Do you have a traceback you can share?

[nmccormick-bombas]

@luandy64 logs below:

2019-12-18 16:49:17,458Z main - INFO Running tap-typeform version 1.1.3 and target-stitch version 3.0.1 2019-12-18 16:49:17,526Z main - INFO Starting tap to discover schemas: tap-env/bin/tap-typeform --config /tmp/tap_discover_config.json --discover 2019-12-18 16:49:17,766Z main - INFO Tap exited normally. 2019-12-18 16:49:17,766Z main - INFO Saving list of discovered streams 2019-12-18 16:49:17,781Z main - INFO Saving structure of stream landings 2019-12-18 16:49:17,833Z main - INFO Saving structure of stream answers 2019-12-18 16:49:17,845Z main - INFO Saving structure of stream questions 2019-12-18 16:49:17,866Z main - INFO Writing catalog to file 2019-12-18 16:49:17,869Z main - INFO Selected fields for stream questions: question_id, title, ref, form_id 2019-12-18 16:49:17,869Z main - INFO Selected fields for stream landings: platform, hidden, landing_id, referer, landed_at, token, user_agent, browser, network_id, submitted_at 2019-12-18 16:49:17,869Z main - INFO Selected fields for stream answers: question_id, landing_id, data_type, type, ref, answer 2019-12-18 16:49:17,872Z main - INFO Current state: {"bookmarks": {"p39mFb": {"date_to_resume": "2019-12-19 00:00:00"}, "ZeKnyY": {"date_to_resume": "2019-12-19 00:00:00"}, "DiCqtz": {"date_to_resume": "2019-12-19 00:00:00"}, "D3rj2g": {"date_to_resume": "2019-11-19 00:00:00"}, "vGOLMy": {"date_to_resume": "2019-12-19 00:00:00"}, "questions": {}, "WNa2yT": {"date_to_resume": "2019-11-19 00:00:00"}, "e5PAt4": {"date_to_resume": "2019-11-19 00:00:00"}, "MVbzki": {"date_to_resume": "2019-12-19 00:00:00"}, "answers": {}, "OICqXr": {"date_to_resume": "2019-12-19 00:00:00"}, "G2f5gR": {"date_to_resume": "2019-12-19 00:00:00"}, "landings": {}}} 2019-12-18 16:49:17,873Z main - INFO Starting tap: tap-env/bin/tap-typeform --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json 2019-12-18 16:49:17,875Z main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json 2019-12-18 16:49:18,064Z tap - INFO form: p39mFb 2019-12-18 16:49:18,245Z tap - INFO "form definition p39mFb" job took 0.2s, ended with status "succeeded" 2019-12-18 16:49:18,246Z tap - INFO replicated 9 records from "questions" endpoint 2019-12-18 16:49:18,246Z tap - INFO start_date: 2017-06-01T00:00:00+00:00 2019-12-18 16:49:18,246Z tap - INFO end_date: 2019-12-19T00:00:00+00:00 2019-12-18 16:49:18,246Z tap - INFO last_date: 2019-12-19T00:00:00+00:00 2019-12-18 16:49:18,246Z tap - INFO ut_current_date: 1576713600 2019-12-18 16:49:18,246Z tap - INFO ut_next_date: 1576800000 2019-12-18 16:49:18,247Z tap - INFO Forms query - form: p39mFb start_date: 2019-12-19 00:00 end_date: 2019-12-20 00:00 2019-12-18 16:49:18,289Z target - INFO Using batch_size_prefernces of {'user_batch_size_preference': None, 'full_table_streams': [], 'batch_size_preference': None} 2019-12-18 16:49:18,290Z target - INFO Using turbo_boost_factor of 1 2019-12-18 16:49:18,552Z tap - INFO raw data items= 0 2019-12-18 16:49:18,552Z tap - INFO "form p39mFb" job took 0.3s, ended with status "succeeded" 2019-12-18 16:49:18,553Z tap - INFO replicated 0 records from "landings" endpoint 2019-12-18 16:49:18,554Z tap - INFO replicated 0 records from "answers" endpoint 2019-12-18 16:49:18,555Z tap - CRITICAL year is out of range 2019-12-18 16:49:18,555Z tap - Traceback (most recent call last): 2019-12-18 16:49:18,555Z tap - File "tap-env/bin/tap-typeform", line 8, in 2019-12-18 16:49:18,556Z tap - sys.exit(main()) 2019-12-18 16:49:18,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 225, in wrapped 2019-12-18 16:49:18,556Z tap - return fnc(*args, kwargs) 2019-12-18 16:49:18,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_typeform/init.py", line 95, in main 2019-12-18 16:49:18,556Z tap - sync(atx) 2019-12-18 16:49:18,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_typeform/init.py", line 76, in sync 2019-12-18 16:49:18,556Z tap - streams.sync_forms(atx) 2019-12-18 16:49:18,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_typeform/streams.py", line 273, in sync_forms 2019-12-18 16:49:18,556Z tap - write_forms_state(atx, form_id, pendulum.parse(max_submitted_at)) 2019-12-18 16:49:18,556Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/pendulum/parser.py", line 20, in parse 2019-12-18 16:49:18,557Z tap - return _parse(text, options) 2019-12-18 16:49:18,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/pendulum/parser.py", line 36, in _parse 2019-12-18 16:49:18,557Z tap - parsed = base_parse(text, options) 2019-12-18 16:49:18,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/pendulum/parsing/init.py", line 70, in parse 2019-12-18 16:49:18,557Z tap - return _normalize(_parse(text, _options), _options) 2019-12-18 16:49:18,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/pendulum/parsing/init.py", line 116, in _parse 2019-12-18 16:49:18,557Z tap - return _parse_common(text, options) 2019-12-18 16:49:18,557Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/pendulum/parsing/init.py", line 173, in _parse_common 2019-12-18 16:49:18,557Z tap - return date(year, month, day) 2019-12-18 16:49:18,557Z tap - ValueError: year is out of range 2019-12-18 16:49:18,577Z target - INFO Sending batch with 9 messages for table questions to https://api.stitchdata.com/v2/import/batch 2019-12-18 16:49:18,706Z target - INFO Requests complete, stopping loop 2019-12-18 16:49:18,753Z main - INFO Target exited normally with status 0 2019-12-18 16:49:18,755Z main - INFO No tunnel subprocess to tear down 2019-12-18 16:49:18,755Z main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "year is out of range". Target succeeded.

[luandy64]

My initial hunch is that max_submitted_at is sometimes None. So we would need to pick some other field as the bookmark

[luandy64]

I've reverted the code to the previous working version. Please close this issue if it's not an issue still happening

[nmccormick-bombas]

Awesome - that worked... thanks so much!