event_by_corp_filing errors processing records from LEAR

[WadeBarnes]

Pipeline Logs:

INFO:bcreg.bcreg_core:Caching data for parties and events ...
INFO:bcreg.bcreg_core:Caching data for corporations ...
INFO:bcreg.bcreg_core:Caching data for code tables ...
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,False
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,True
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    sort version records for: FM0794306
INFO:bcreg.bcreg_lear:    done.
INFO:bcreg.bcreg_lear:>>> get corp info for: PA0000002,False
INFO:bcreg.bcreg_lear:No corp rec found for PA0000002
ERROR:bcreg.bcreg_lear:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.bcreg_lear:None
ERROR:bcreg.eventprocessor:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 1929, in process_corp_event_queue_internal
    corp_info_json = bc_registries.to_json(corp_info)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 906, in to_json
    ret = json.dumps(data, cls=CustomJsonEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 94, in default
    raise error
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.eventprocessor:None
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,False
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,True
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    sort version records for: FM0794306
INFO:bcreg.bcreg_lear:    done.
INFO:bcreg.bcreg_lear:>>> get corp info for: PA0000002,False
INFO:bcreg.bcreg_lear:No corp rec found for PA0000002
ERROR:bcreg.bcreg_lear:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.bcreg_lear:None
ERROR:bcreg.eventprocessor:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 1929, in process_corp_event_queue_internal
    corp_info_json = bc_registries.to_json(corp_info)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 906, in to_json
    ret = json.dumps(data, cls=CustomJsonEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 94, in default
    raise error
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.eventprocessor:None

Pipeline Status Records:

[
    {
        'record_id': 10639773, 
        'system_type_cd': 'BCREG_LEAR', 
        'corp_num': 'FM0794306', 
        'prev_event_id': 1648671, 
        'prev_event_date': datetime.datetime(2023, 6, 12, 22, 44, 23, 777035), 
        'last_event_id': 1648680, 
        'last_event_date': datetime.datetime(2023, 6, 12, 23, 1, 19, 175234), 
        'entry_date': datetime.datetime(2023, 6, 12, 23, 15, 1, 96251), 
        'process_date': datetime.datetime(2023, 6, 20, 14, 50, 34, 438742), 
        'process_success': 'N', 
        'process_msg': 'year 0 is out of range'
    }, 
    {
        'record_id': 10639628, 
        'system_type_cd': 'BCREG_LEAR', 
        'corp_num': 'FM0794306', 
        'prev_event_id': 1648645, 
        'prev_event_date': datetime.datetime(2023, 6, 12, 22, 10, 41, 927270), 
        'last_event_id': 1648671, 
        'last_event_date': datetime.datetime(2023, 6, 12, 22, 44, 23, 777035), 
        'entry_date': datetime.datetime(2023, 6, 12, 22, 45, 1, 217412), 
        'process_date': datetime.datetime(2023, 6, 20, 14, 50, 34, 423968), 
        'process_success': 'N', 
        'process_msg': 'year 0 is out of range'
    }
]

cc @esune, @amanji, @ianco

[swcurran]

This is a problem with 2 records? Are the rest processing OK? Any other information beyond the logs to provide, @WadeBarnes ?

BC Registries has in the past used a date of 0 to trigger magic handling, so my guess is this is an instance of that.

@thorwolpert — could you please raise a flag with other BC Registries folks about this? I don’t know/can’t find the GitHub ID of folks like Dwayne and David Roberts.

[WadeBarnes]

Yes, just an issue with the two records. All others are fine. No additional information yet. I have not looked at the data or associated code yet.

[swcurran]

OK — I think the next step is to have BC Reg look at this entity (it’s one sole prop) on their side and see what is magic about it. Their registration is in OrgBook BC — https://orgbook.gov.bc.ca/entity/FM0794306

I don’t see any new records associated with the entity, which is what is implied with these process failures.

[WadeBarnes]

BTW, this issue is unrelated to https://github.com/bcgov/aries-vcr/issues/751 which was causing credential processing (posting) errors.

[swcurran]

Email sent to Registries to ask them to look into the issue.

[WadeBarnes]

Code that is failing: https://github.com/bcgov/von-bc-registries-agent/blob/ee1aa1e942585fcd9a7cedde600d463bfc3067da/data-pipeline/bcreg/bcreg_lear.py#L85-L94

[WadeBarnes]

Based on feedback from BC Registries PA0000002 has all it's dates set to 0001-01-01 which cause the error we're seeing. It's a little strange though, that the code is flagging the records for FM0794306 and not PA0000002 with the error.

We are waiting for a response from BC Registries as to why all the dates are set to 0001-01-01 on PA0000002.

[esune]

@WadeBarnes @swcurran any updates on this? Can we close the issue if the problem is with the data, or do we still have something we need to be looking into?

[WadeBarnes]

I forwarded you some additional information. Some decisions need to be made as to how we handle these situations. It looks like the BC Reg code may handle these issues, but the LEAR code does not contain the same logic.

[WadeBarnes]

@esune, have we looked into fixing this issue yet?

[esune]

I believe this was being tackled by @amanji together with the other fixes to the pipeline - see https://app.zenhub.com/workspaces/bcgov-ditp-products-team-63c987121278d5001ecb177b/issues/zh/41

I believe the changes have been merged and are waiting for validation and promotion.

[WadeBarnes]

@amanji, is there a PR associated to this ticket?

[esune]

Checked-in with @amanji, this is still an issue. We might want to deal with it as part of the rewrite in #334