search

singer-io / tap-zoom

GNU Affero General Public License v3.0
4 stars 15 forks source link

Schema Violation errors during extraction #3

Open dsprayberry opened 4 years ago

dsprayberry commented 4 years ago

A Zoom integration running tap-zoom version 0.0.1 encountered schema violations during transformation, seemingly from the report_webinars stream based on the included fields, though the affected stream is unclear from the exception. Selected streams for this connection include:

Full traceback:

2020-09-14 17:01:04,910Z tap - Traceback (most recent call last): 2020-09-14 17:01:04,911Z tap - File "tap-env/bin/tap-zoom", line 33, in 2020-09-14 17:01:04,911Z tap - sys.exit(load_entry_point('tap-zoom==0.0.1', 'console_scripts', 'tap-zoom')()) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped 2020-09-14 17:01:04,911Z tap - return fnc(*args, **kwargs) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/init.py", line 41, in main 2020-09-14 17:01:04,911Z tap - parsed_args.state) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 138, in sync 2020-09-14 17:01:04,911Z tap - {}) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 89, in sync_endpoint 2020-09-14 17:01:04,911Z tap - child_key_bag) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 89, in sync_endpoint 2020-09-14 17:01:04,911Z tap - child_key_bag) 2020-09-14 17:01:04,911Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 72, in sync_endpoint 2020-09-14 17:01:04,911Z tap - mdata) 2020-09-14 17:01:04,912Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 140, in transform 2020-09-14 17:01:04,912Z tap - raise SchemaMismatch(self.errors) 2020-09-14 17:01:04,912Z tap - singer.transform.SchemaMismatch: Errors during transform 2020-09-14 17:01:04,912Z tap - dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'} 2020-09-14 17:01:04,912Z tap - : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}} 2020-09-14 17:01:04,912Z tap - 2020-09-14 17:01:04,912Z tap - 2020-09-14 17:01:04,912Z tap - Errors during transform: [dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}}]

Full extraction logs in case they're helpful:


2020-09-14 17:00:49,418Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true )
2020-09-14 17:00:49,423Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true )
2020-09-14 17:00:50,433Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [32] at offset None
2020-09-14 17:00:50,434Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-zoom --config /tmp/tap_discover_config.json --discover
2020-09-14 17:00:50,752Z    tap - INFO Testing authentication
2020-09-14 17:00:50,812Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.05941438674926758, tags={'http_status_code': 200, 'status': 'succeeded'})
2020-09-14 17:00:50,813Z    tap - INFO Starting discover
2020-09-14 17:00:50,828Z    tap - INFO Finished discover
2020-09-14 17:00:50,885Z   main - INFO Tap exited normally.
2020-09-14 17:00:50,887Z   main - INFO Saving list of discovered streams
2020-09-14 17:00:54,008Z   main - INFO Saving structure of stream report_meeting_participants (tap_stream_id: report_meeting_participants)
2020-09-14 17:00:55,302Z   main - INFO Saving structure of stream webinar_polls (tap_stream_id: webinar_polls)
2020-09-14 17:00:55,353Z   main - INFO Saving structure of stream webinar_qna_results (tap_stream_id: webinar_qna_results)
2020-09-14 17:00:55,865Z   main - INFO Saving structure of stream webinar_tracking_sources (tap_stream_id: webinar_tracking_sources)
2020-09-14 17:00:55,937Z   main - INFO Saving structure of stream report_meetings (tap_stream_id: report_meetings)
2020-09-14 17:00:55,985Z   main - INFO Saving structure of stream meeting_questions (tap_stream_id: meeting_questions)
2020-09-14 17:00:56,025Z   main - INFO Saving structure of stream users (tap_stream_id: users)
2020-09-14 17:00:56,081Z   main - INFO Saving structure of stream webinar_questions (tap_stream_id: webinar_questions)
2020-09-14 17:00:56,125Z   main - INFO Saving structure of stream webinar_registrants (tap_stream_id: webinar_registrants)
2020-09-14 17:00:56,173Z   main - INFO Saving structure of stream meeting_files (tap_stream_id: meeting_files)
2020-09-14 17:00:56,248Z   main - INFO Saving structure of stream webinar_absentees (tap_stream_id: webinar_absentees)
2020-09-14 17:00:56,312Z   main - INFO Saving structure of stream webinar_poll_results (tap_stream_id: webinar_poll_results)
2020-09-14 17:00:56,368Z   main - INFO Saving structure of stream meeting_poll_results (tap_stream_id: meeting_poll_results)
2020-09-14 17:00:56,431Z   main - INFO Saving structure of stream report_webinar_participants (tap_stream_id: report_webinar_participants)
2020-09-14 17:00:56,570Z   main - INFO Saving structure of stream meeting_registrants (tap_stream_id: meeting_registrants)
2020-09-14 17:00:56,626Z   main - INFO Saving structure of stream report_webinars (tap_stream_id: report_webinars)
2020-09-14 17:00:56,731Z   main - INFO Saving structure of stream meetings (tap_stream_id: meetings)
2020-09-14 17:00:58,691Z   main - INFO Saving structure of stream webinar_panelists (tap_stream_id: webinar_panelists)
2020-09-14 17:00:58,812Z   main - INFO Saving structure of stream webinars (tap_stream_id: webinars)
2020-09-14 17:00:59,011Z   main - INFO Saving structure of stream meeting_polls (tap_stream_id: meeting_polls)
2020-09-14 17:00:59,072Z   main - INFO Saving structure of stream webinar_files (tap_stream_id: webinar_files)
2020-09-14 17:00:59,129Z   main - INFO Writing catalog to file
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinars: id, agenda, timezone, created_at, occurences, start_url, duration, tracking_fields, start_time, settings, uuid, recurrence, join_url, topic, type, host_id
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_poll_results: email, webinar_uuid, question_details, name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_absentees: address, zip, org, state, comments, status, county, job_title, no_of_employees, create_time, industry, id, first_name, purchasing_time_frame, join_url, city, phone, custom_questions, email, webinar_uuid, role_in_purchase_process, last_name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_polls: id, questions, webinar_id, status, title
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream report_webinars: id, participants_count, user_email, user_name, topic, duration, tracking_fields, start_time, uuid, dept, webinar_id, end_time, type, total_minutes
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_questions: custom_questions, questions, webinar_id
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream users: id, pmi, timezone, first_name, dept, im_group_ids, last_login_time, created_at, last_client_version, status, email, type, group_ids, last_name, verified
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_qna_results: email, webinar_uuid, question_details, name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_registrants: address, zip, state, comments, status, county, job_title, no_of_employees, create_time, industry, id, first_name, purchasing_time_frame, join_url, city, webinar_id, phone, custom_questions, email, org, role_in_purchase_process, last_name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_files: download_url, file_size, webinar_uuid, file_name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_panelists: id, email, join_url, webinar_id, name
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream webinar_tracking_sources: id, webinar_id, source_name, tracking_url, visitor_count, registration_count
2020-09-14 17:00:59,153Z   main - INFO Selected fields for stream report_webinar_participants: id, attentiveness_score, webinar_id, leave_time, user_email, join_time, duration, name, user_id
2020-09-14 17:00:59,258Z   main - INFO Current state: {"currently_syncing": "users"}
2020-09-14 17:00:59,259Z   main - INFO Starting tap: tap-env/bin/tap-zoom --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2020-09-14 17:00:59,263Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meeting_files
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meeting_poll_results
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meeting_registrants
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meetings
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meeting_polls
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: report_meeting_participants
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: report_meetings
2020-09-14 17:00:59,616Z    tap - INFO Skipping stream: meeting_questions
2020-09-14 17:00:59,710Z    tap - INFO HTTP request to "users" endpoint took 0.092s, returned status code 200
2020-09-14 17:00:59,738Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.024764299392700195, tags={'endpoint': 'list_webinars', 'status': 'succeeded'})
2020-09-14 17:00:59,758Z target - INFO Using batch_size_prefernces of {'user_batch_size_preference': None, 'full_table_streams': [], 'batch_size_preference': None}
2020-09-14 17:00:59,758Z   main - INFO State update: adding currently_syncing = "users"
2020-09-14 17:00:59,764Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.024234294891357422, tags={'endpoint': 'list_webinars', 'status': 'succeeded'})
2020-09-14 17:00:59,793Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.02692556381225586, tags={'endpoint': 'list_webinars', 'status': 'succeeded'})
2020-09-14 17:00:59,823Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.02739691734313965, tags={'endpoint': 'list_webinars', 'status': 'succeeded'})
2020-09-14 17:00:59,859Z    tap - INFO HTTP request to "list_webinars" endpoint took 0.034s, returned status code 200
2020-09-14 17:00:59,860Z target - INFO Serializing batch with 5 messages for table users
2020-09-14 17:00:59,861Z target - INFO Sending batch of 4045 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:00:59,888Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.027252912521362305, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:00:59,938Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.049s, returned status code 200
2020-09-14 17:00:59,940Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:00:59,940Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:00:59,940Z target - INFO Sending batch of 2040 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:00:59,978Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.037s, returned status code 200
2020-09-14 17:00:59,980Z    tap - INFO replicated 4 records from "webinar_panelists" endpoint
2020-09-14 17:00:59,980Z target - INFO Serializing batch with 4 messages for table webinar_panelists
2020-09-14 17:00:59,980Z target - INFO Sending batch of 1946 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,010Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.029s, returned status code 200
2020-09-14 17:01:00,011Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:00,066Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.054s, returned status code 200
2020-09-14 17:01:00,068Z    tap - INFO replicated 3 records from "report_webinar_participants" endpoint
2020-09-14 17:01:00,068Z target - INFO Serializing batch with 3 messages for table report_webinar_participants
2020-09-14 17:01:00,068Z target - INFO Sending batch of 2078 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,104Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.035s, returned status code 200
2020-09-14 17:01:00,106Z    tap - INFO replicated 0 records from "webinar_registrants" endpoint
2020-09-14 17:01:00,136Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.030s, returned status code 200
2020-09-14 17:01:00,138Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:00,198Z    tap - INFO HTTP request to "webinars" endpoint took 0.058s, returned status code 200
2020-09-14 17:01:00,200Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:00,201Z target - INFO Sending batch of 7257 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,236Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.034540414810180664, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:00,278Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04098081588745117, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:00,319Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03930187225341797, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:00,356Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03463482856750488, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:00,356Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:00,397Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04013514518737793, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:00,460Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.061s, returned status code 200
2020-09-14 17:01:00,461Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:00,461Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:00,462Z target - INFO Sending batch of 2041 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,501Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.039s, returned status code 200
2020-09-14 17:01:00,503Z    tap - INFO replicated 5 records from "webinar_panelists" endpoint
2020-09-14 17:01:00,503Z target - INFO Serializing batch with 5 messages for table webinar_panelists
2020-09-14 17:01:00,504Z target - INFO Sending batch of 2248 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,536Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.031s, returned status code 200
2020-09-14 17:01:00,537Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:00,612Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.074s, returned status code 200
2020-09-14 17:01:00,614Z    tap - INFO replicated 2 records from "report_webinar_participants" endpoint
2020-09-14 17:01:00,615Z target - INFO Serializing batch with 2 messages for table report_webinar_participants
2020-09-14 17:01:00,615Z target - INFO Sending batch of 1724 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,650Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.035s, returned status code 200
2020-09-14 17:01:00,651Z    tap - INFO replicated 0 records from "webinar_registrants" endpoint
2020-09-14 17:01:00,694Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.042s, returned status code 200
2020-09-14 17:01:00,695Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:00,802Z    tap - INFO HTTP request to "webinars" endpoint took 0.105s, returned status code 200
2020-09-14 17:01:00,804Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:00,805Z target - INFO Sending batch of 6865 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:00,842Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03666114807128906, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:00,876Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03324532508850098, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:00,911Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03400540351867676, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:00,944Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03163313865661621, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:00,944Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:00,972Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.027176856994628906, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:01,029Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.055s, returned status code 200
2020-09-14 17:01:01,030Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:01,031Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:01,031Z target - INFO Sending batch of 2035 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,066Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.034s, returned status code 200
2020-09-14 17:01:01,068Z    tap - INFO replicated 3 records from "webinar_panelists" endpoint
2020-09-14 17:01:01,068Z target - INFO Serializing batch with 3 messages for table webinar_panelists
2020-09-14 17:01:01,068Z target - INFO Sending batch of 1437 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,102Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.031s, returned status code 200
2020-09-14 17:01:01,102Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:01,256Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.153s, returned status code 200
2020-09-14 17:01:01,259Z    tap - INFO replicated 3 records from "report_webinar_participants" endpoint
2020-09-14 17:01:01,260Z target - INFO Serializing batch with 3 messages for table report_webinar_participants
2020-09-14 17:01:01,260Z target - INFO Sending batch of 2075 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,293Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.031s, returned status code 200
2020-09-14 17:01:01,294Z    tap - INFO replicated 0 records from "webinar_registrants" endpoint
2020-09-14 17:01:01,326Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.032s, returned status code 200
2020-09-14 17:01:01,327Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:01,429Z    tap - INFO HTTP request to "webinars" endpoint took 0.100s, returned status code 200
2020-09-14 17:01:01,431Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:01,431Z target - INFO Sending batch of 7185 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,466Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03422379493713379, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:01,505Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03745007514953613, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:01,547Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04071784019470215, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:01,581Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03206801414489746, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:01,581Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:01,612Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.02992415428161621, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:01,667Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.054s, returned status code 200
2020-09-14 17:01:01,670Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:01,670Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:01,670Z target - INFO Sending batch of 2045 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,714Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.043s, returned status code 200
2020-09-14 17:01:01,715Z    tap - INFO replicated 5 records from "webinar_panelists" endpoint
2020-09-14 17:01:01,715Z target - INFO Serializing batch with 5 messages for table webinar_panelists
2020-09-14 17:01:01,716Z target - INFO Sending batch of 2241 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,751Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.035s, returned status code 200
2020-09-14 17:01:01,752Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:01,825Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.072s, returned status code 200
2020-09-14 17:01:01,831Z    tap - INFO replicated 7 records from "report_webinar_participants" endpoint
2020-09-14 17:01:01,832Z target - INFO Serializing batch with 7 messages for table report_webinar_participants
2020-09-14 17:01:01,832Z target - INFO Sending batch of 3525 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:01,875Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.042s, returned status code 200
2020-09-14 17:01:01,876Z    tap - INFO replicated 0 records from "webinar_registrants" endpoint
2020-09-14 17:01:01,910Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.032s, returned status code 200
2020-09-14 17:01:01,911Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:01,998Z    tap - INFO HTTP request to "webinars" endpoint took 0.084s, returned status code 200
2020-09-14 17:01:02,001Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:02,002Z target - INFO Sending batch of 6984 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,048Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04633522033691406, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:02,092Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03837466239929199, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:02,131Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04166722297668457, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:02,191Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.05846238136291504, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:02,192Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:02,319Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.12621212005615234, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:02,379Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:02,380Z target - INFO Sending batch of 2020 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,380Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.053s, returned status code 200
2020-09-14 17:01:02,388Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:02,425Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.042s, returned status code 200
2020-09-14 17:01:02,427Z    tap - INFO replicated 9 records from "webinar_panelists" endpoint
2020-09-14 17:01:02,429Z target - INFO Serializing batch with 9 messages for table webinar_panelists
2020-09-14 17:01:02,430Z target - INFO Sending batch of 3132 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,472Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.036s, returned status code 200
2020-09-14 17:01:02,473Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:02,589Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.115s, returned status code 200
2020-09-14 17:01:02,636Z    tap - INFO replicated 73 records from "report_webinar_participants" endpoint
2020-09-14 17:01:02,637Z target - INFO Serializing batch with 73 messages for table report_webinar_participants
2020-09-14 17:01:02,637Z target - INFO Sending batch of 27355 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,678Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.039s, returned status code 200
2020-09-14 17:01:02,681Z    tap - INFO replicated 3 records from "webinar_registrants" endpoint
2020-09-14 17:01:02,682Z target - INFO Serializing batch with 3 messages for table webinar_registrants
2020-09-14 17:01:02,682Z target - INFO Sending batch of 4418 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,721Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.037s, returned status code 200
2020-09-14 17:01:02,722Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:02,810Z    tap - INFO HTTP request to "webinars" endpoint took 0.086s, returned status code 200
2020-09-14 17:01:02,817Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:02,817Z target - INFO Sending batch of 7089 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:02,857Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03782916069030762, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:02,897Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.037835121154785156, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:02,935Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03632235527038574, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:02,971Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.033721923828125, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:02,971Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:03,002Z    tap - INFO HTTP request to "webinar_questions" endpoint took 0.028s, returned status code 200
2020-09-14 17:01:03,004Z    tap - INFO replicated 1 records from "webinar_questions" endpoint
2020-09-14 17:01:03,004Z target - INFO Serializing batch with 1 messages for table webinar_questions
2020-09-14 17:01:03,004Z target - INFO Sending batch of 1001 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,038Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03314614295959473, tags={'endpoint': 'report_webinars', 'status': 'succeeded'})
2020-09-14 17:01:03,070Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.030s, returned status code 200
2020-09-14 17:01:03,071Z    tap - INFO replicated 0 records from "webinar_panelists" endpoint
2020-09-14 17:01:03,112Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.040s, returned status code 200
2020-09-14 17:01:03,114Z    tap - INFO replicated 0 records from "webinar_polls" endpoint
2020-09-14 17:01:03,151Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03556036949157715, tags={'endpoint': 'report_webinar_participants', 'status': 'succeeded'})
2020-09-14 17:01:03,198Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.044s, returned status code 200
2020-09-14 17:01:03,199Z    tap - INFO replicated 0 records from "webinar_registrants" endpoint
2020-09-14 17:01:03,246Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.045s, returned status code 200
2020-09-14 17:01:03,248Z    tap - INFO replicated 1 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:03,251Z target - INFO Serializing batch with 1 messages for table webinar_tracking_sources
2020-09-14 17:01:03,251Z target - INFO Sending batch of 980 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,309Z    tap - INFO HTTP request to "webinars" endpoint took 0.057s, returned status code 200
2020-09-14 17:01:03,311Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:03,312Z target - INFO Sending batch of 5654 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,350Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.038100481033325195, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:03,392Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.040392398834228516, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:03,426Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.032204627990722656, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:03,460Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.0328984260559082, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:03,461Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:03,502Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03985595703125, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:03,616Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.112s, returned status code 200
2020-09-14 17:01:03,618Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:03,618Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:03,619Z target - INFO Sending batch of 2016 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,654Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.034s, returned status code 200
2020-09-14 17:01:03,655Z    tap - INFO replicated 3 records from "webinar_panelists" endpoint
2020-09-14 17:01:03,656Z target - INFO Serializing batch with 3 messages for table webinar_panelists
2020-09-14 17:01:03,656Z target - INFO Sending batch of 1437 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,702Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.044s, returned status code 200
2020-09-14 17:01:03,704Z    tap - INFO replicated 1 records from "webinar_polls" endpoint
2020-09-14 17:01:03,706Z target - INFO Serializing batch with 1 messages for table webinar_polls
2020-09-14 17:01:03,706Z target - INFO Sending batch of 1133 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,790Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.083s, returned status code 200
2020-09-14 17:01:03,794Z    tap - INFO replicated 2 records from "report_webinar_participants" endpoint
2020-09-14 17:01:03,795Z target - INFO Serializing batch with 2 messages for table report_webinar_participants
2020-09-14 17:01:03,795Z target - INFO Sending batch of 1704 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,840Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.043s, returned status code 200
2020-09-14 17:01:03,841Z    tap - INFO replicated 1 records from "webinar_registrants" endpoint
2020-09-14 17:01:03,841Z target - INFO Serializing batch with 1 messages for table webinar_registrants
2020-09-14 17:01:03,842Z target - INFO Sending batch of 2742 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,875Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.033s, returned status code 200
2020-09-14 17:01:03,876Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:03,938Z    tap - INFO HTTP request to "webinars" endpoint took 0.059s, returned status code 200
2020-09-14 17:01:03,942Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:03,942Z target - INFO Sending batch of 5862 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:03,986Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04186844825744629, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:04,016Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.028987884521484375, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:04,051Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03362083435058594, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:04,088Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03531694412231445, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:04,088Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:04,118Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.029132843017578125, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:04,204Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.085s, returned status code 200
2020-09-14 17:01:04,206Z    tap - INFO replicated 1 records from "report_webinars" endpoint
2020-09-14 17:01:04,206Z target - INFO Serializing batch with 1 messages for table report_webinars
2020-09-14 17:01:04,206Z target - INFO Sending batch of 2015 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,249Z    tap - INFO HTTP request to "webinar_panelists" endpoint took 0.041s, returned status code 200
2020-09-14 17:01:04,250Z    tap - INFO replicated 4 records from "webinar_panelists" endpoint
2020-09-14 17:01:04,251Z target - INFO Serializing batch with 4 messages for table webinar_panelists
2020-09-14 17:01:04,251Z target - INFO Sending batch of 1727 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,358Z    tap - INFO HTTP request to "webinar_polls" endpoint took 0.104s, returned status code 200
2020-09-14 17:01:04,359Z    tap - INFO replicated 1 records from "webinar_polls" endpoint
2020-09-14 17:01:04,360Z target - INFO Serializing batch with 1 messages for table webinar_polls
2020-09-14 17:01:04,360Z target - INFO Sending batch of 1097 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,459Z    tap - INFO HTTP request to "report_webinar_participants" endpoint took 0.098s, returned status code 200
2020-09-14 17:01:04,466Z    tap - INFO replicated 17 records from "report_webinar_participants" endpoint
2020-09-14 17:01:04,467Z target - INFO Serializing batch with 17 messages for table report_webinar_participants
2020-09-14 17:01:04,468Z target - INFO Sending batch of 7110 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,504Z    tap - INFO HTTP request to "webinar_registrants" endpoint took 0.036s, returned status code 200
2020-09-14 17:01:04,507Z    tap - INFO replicated 3 records from "webinar_registrants" endpoint
2020-09-14 17:01:04,508Z target - INFO Serializing batch with 3 messages for table webinar_registrants
2020-09-14 17:01:04,508Z target - INFO Sending batch of 3728 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,546Z    tap - INFO HTTP request to "webinar_tracking_sources" endpoint took 0.035s, returned status code 200
2020-09-14 17:01:04,547Z    tap - INFO replicated 0 records from "webinar_tracking_sources" endpoint
2020-09-14 17:01:04,623Z    tap - INFO HTTP request to "webinars" endpoint took 0.074s, returned status code 200
2020-09-14 17:01:04,627Z target - INFO Serializing batch with 1 messages for table webinars
2020-09-14 17:01:04,627Z target - INFO Sending batch of 5910 bytes to https://api.stitchdata.com/v2/import/batch
2020-09-14 17:01:04,664Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.036644935607910156, tags={'endpoint': 'webinar_absentees', 'status': 'succeeded'})
2020-09-14 17:01:04,725Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.059514522552490234, tags={'endpoint': 'webinar_poll_results', 'status': 'succeeded'})
2020-09-14 17:01:04,775Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.04863119125366211, tags={'endpoint': 'webinar_qna_results', 'status': 'succeeded'})
2020-09-14 17:01:04,811Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03459978103637695, tags={'endpoint': 'webinar_files', 'status': 'succeeded'})
2020-09-14 17:01:04,811Z    tap - INFO replicated 1 records from "webinars" endpoint
2020-09-14 17:01:04,846Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=0.03328680992126465, tags={'endpoint': 'webinar_questions', 'status': 'succeeded'})
2020-09-14 17:01:04,905Z    tap - INFO HTTP request to "report_webinars" endpoint took 0.058s, returned status code 200
2020-09-14 17:01:04,907Z    tap - INFO replicated 0 records from "report_webinars" endpoint
2020-09-14 17:01:04,908Z    tap - INFO replicated 0 records from "list_webinars" endpoint
2020-09-14 17:01:04,908Z    tap - INFO replicated 5 records from "users" endpoint
2020-09-14 17:01:04,909Z    tap - CRITICAL Errors during transform
2020-09-14 17:01:04,910Z    tap -   dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}
2020-09-14 17:01:04,910Z    tap -   : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}}
2020-09-14 17:01:04,910Z    tap - 
2020-09-14 17:01:04,910Z    tap - 
2020-09-14 17:01:04,910Z    tap - Errors during transform: [dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}}]
2020-09-14 17:01:04,910Z    tap - Traceback (most recent call last):
2020-09-14 17:01:04,911Z    tap -   File "tap-env/bin/tap-zoom", line 33, in <module>
2020-09-14 17:01:04,911Z    tap -     sys.exit(load_entry_point('tap-zoom==0.0.1', 'console_scripts', 'tap-zoom')())
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2020-09-14 17:01:04,911Z    tap -     return fnc(*args, **kwargs)
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/__init__.py", line 41, in main
2020-09-14 17:01:04,911Z    tap -     parsed_args.state)
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 138, in sync
2020-09-14 17:01:04,911Z    tap -     {})
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 89, in sync_endpoint
2020-09-14 17:01:04,911Z    tap -     child_key_bag)
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 89, in sync_endpoint
2020-09-14 17:01:04,911Z    tap -     child_key_bag)
2020-09-14 17:01:04,911Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_zoom/sync.py", line 72, in sync_endpoint
2020-09-14 17:01:04,911Z    tap -     mdata)
2020-09-14 17:01:04,912Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 140, in transform
2020-09-14 17:01:04,912Z    tap -     raise SchemaMismatch(self.errors)
2020-09-14 17:01:04,912Z    tap - singer.transform.SchemaMismatch: Errors during transform
2020-09-14 17:01:04,912Z    tap -   dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}
2020-09-14 17:01:04,912Z    tap -   : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}}
2020-09-14 17:01:04,912Z    tap - 
2020-09-14 17:01:04,912Z    tap - 
2020-09-14 17:01:04,912Z    tap - Errors during transform: [dept: data does not match {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, : data does not match {'type': 'object', 'selected': True, 'additionalProperties': False, 'properties': {'id': {'type': ['integer'], 'selected': True, 'inclusion': 'available'}, 'user_email': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'dept': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'tracking_fields': {'type': ['array', 'null'], 'selected': True, 'items': {'type': ['object'], 'additionalProperties': False, 'properties': {'value': {'type': ['null', 'string']}, 'field': {'type': ['null', 'string']}}}, 'inclusion': 'available'}, 'webinar_id': {'type': ['string'], 'selected': True, 'inclusion': 'available'}, 'duration': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'total_minutes': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'type': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}, 'start_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'topic': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'end_time': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available', 'format': 'date-time'}, 'user_name': {'type': ['string', 'null'], 'selected': True, 'inclusion': 'available'}, 'uuid': {'type': ['string'], 'selected': True, 'inclusion': 'automatic'}, 'participants_count': {'type': ['integer', 'null'], 'selected': True, 'inclusion': 'available'}}}]
2020-09-14 17:01:04,975Z target - INFO Requests complete, stopping loop
2020-09-14 17:01:05,077Z   main - INFO Target exited normally with status 0
2020-09-14 17:01:05,437Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [33] at offset None
2020-09-14 17:01:05,438Z   main - INFO No tunnel subprocess to tear down
2020-09-14 17:01:05,438Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "Errors during transform". Target succeeded.```
kekoav commented 4 years ago

@dylan-stitch @cmerrick I have a fix in #4 , I tested the tap locally.

I did notice there were additional fields from the API not in the schema, I added a couple I saw here, but this should probably be audited for field additions to Zoom's API.

adamluba1 commented 4 years ago

I am experiencing the same error. Are there any plans to merge the fix?

jdotjdot commented 4 years ago

This is still an issue, even if you choose not to track the users table. The integration simply doesn't work at the moment.

rdeshmukh15 commented 1 year ago

@kekoav @jdotjdot @adamluba1: Is this issue still persists?