search

cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

[BUG] Signs > AGOL ETL is failing #11636

Closed johnclary closed 1 year ago

johnclary commented 1 year ago

The signs > AGOL ETL is failing with the error:

File "/app/atd-knack-services/services/utils/agol.py", line 84, in handle_response [2023-02-28 19:12:01,416] {{docker_operator.py:242}} INFO - raise ValueError(feature_status["error"]) ValueError: {'code': 1000, 'description': 'No mapping exists from object type System.Numerics.BigInteger to a known managed provider native type.'}

Here's an example Airflow error in slack.

johnclary commented 1 year ago

FYI @chiaberry @ChristinaTremel. chia may have already done some triage?

ChristinaTremel commented 1 year ago

@johnclary - yes, @chiaberry brought this up last week and I saw that she might be working on this today!

chiaberry commented 1 year ago

I found 5 records that are erroring out, still trying to figure out what they have in common. The rest of the records have been updated in agol.

chiaberry commented 1 year ago

What they have in common is a longer DIG_TESS_NUMBER. However, there are older records that also have a long number, so I dont know why it started failing recently. I updated all the records in agol, ETL still not working.

chiaberry commented 1 year ago

We are going to remove the DIG_TESS_NUMBER from the agol dataset, however I couldn't figure out how to delete it and asked Andrew for help.

I think you are unable to delete that field because we have joined views being created off that service, so I just need to remove those joins prior to deleting the field.

He is going to help out when he has the bandwidth, hopefully tomorrow.

chiaberry commented 1 year ago

Andrew removed the field and the ETL is back in green!