[Bug]: Corporate actions support broken

darose commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

In v0.11.0 of alpaca-py when I make the following call to the API:

        ca_request = GetCorporateAnnouncementsRequest(
            since=Utils.to_timestamp(self._prev_trading_date),
            until=Utils.to_timestamp(curr_trading_date),
            ca_types=[t for t in CorporateActionType]
        )
        corporate_actions = self._trading_client.get_corporate_announcements(filter=ca_request)

... I receive the following error messages in response:

  File "/usr/lib/python3.11/site-packages/alpaca/trading/client.py", line 640, in get_corporate_announcements                                                                
    return TypeAdapter(List[CorporateActionAnnouncement]).validate_python(response)                                                                                          
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                          
  File "/usr/lib/python3.11/site-packages/pydantic/type_adapter.py", line 208, in validate_python                                                                            
    return self.validator.validate_python(__object, strict=strict, from_attributes=from_attributes, context=context)                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                         
pydantic_core._pydantic_core.ValidationError: 21 validation errors for list[CorporateActionAnnouncement]                                                                     
1.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': '0c18608a-dec0-4cf...: '25', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
2.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': 'f3d5ee6d-f1bb-493...': '5', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
3.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': 'cbf714cc-f61d-4f5...: '20', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
4.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': '7dacfda6-1fc8-456...': '3', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
5.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': '390267bb-4d0f-427...': '5', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
6.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': 'b8cf48da-5ab9-4e9...71428', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
7.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': 'ff1cd186-9212-4af...: '10', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
8.declaration_date                                                                                                                                                           
  Field required [type=missing, input_value={'id': '5fa9b2e9-7836-41f...: '24', 'new_rate': '1'}, input_type=dict]                                                           
    For further information visit https://errors.pydantic.dev/2.3/v/missing                                                                                                  
9.declaration_date
  Field required [type=missing, input_value={'id': '7df9f1bf-a2a6-46d...': '2', 'new_rate': '1'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
110.declaration_date
  Field required [type=missing, input_value={'id': '98362490-d417-458...: '10', 'new_rate': '1'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
111.declaration_date
  Field required [type=missing, input_value={'id': '929e71e1-72e7-42c...: '20', 'new_rate': '1'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
112.declaration_date
  Field required [type=missing, input_value={'id': 'ad921747-bddb-460...: '35', 'new_rate': '1'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
374.declaration_date
  Field required [type=missing, input_value={'id': 'c2bece15-08e2-403...': '1', 'new_rate': '0'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
374.ex_date
  Field required [type=missing, input_value={'id': 'c2bece15-08e2-403...': '1', 'new_rate': '0'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
374.record_date
  Input should be a valid date [type=date_type, input_value=None, input_type=NoneType] 
    For further information visit https://errors.pydantic.dev/2.3/v/date_type
374.payable_date
  Field required [type=missing, input_value={'id': 'c2bece15-08e2-403...': '1', 'new_rate': '0'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
375.declaration_date
  Field required [type=missing, input_value={'id': '36835cc1-6ebd-428... '1', 'new_rate': '0.5'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
376.declaration_date
  Field required [type=missing, input_value={'id': '078e2f9c-dd2a-4ba..., 'new_rate': '1.17296'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
376.ex_date
  Field required [type=missing, input_value={'id': '078e2f9c-dd2a-4ba..., 'new_rate': '1.17296'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing
376.record_date
  Input should be a valid date [type=date_type, input_value=None, input_type=NoneType] 
    For further information visit https://errors.pydantic.dev/2.3/v/date_type
376.payable_date
  Field required [type=missing, input_value={'id': '078e2f9c-dd2a-4ba..., 'new_rate': '1.17296'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.3/v/missing

Expected Behavior

The API should not display any errors, and should instead return a list of CorporateActionAnnouncement objects.

SDK Version I encountered this issue in

v0.11.0

Steps To Reproduce

1. call .get_corporate_announcements()
2. receive error messages as above

Filled out the Steps to Reproduce section?

[X] I have entered valid steps to reproduce my issue or have attached a minimally reproducible case in code that shows my issue happening; and understand that without this my issue will be flagged as invalid and closed after 30 days.

Anything else?

No response

DougForrest commented 8 months ago

Thanks for the python interface, I've just started using it and encountered the same issue using alpaca-py==0.14.0, where Pydantic validation fails due to missing fields in the API response. I believe this problem results from Pydantic's handling of optional fields, see Pydantic v2 migration guide.

In the guide, it's mentioned that class attributes defined as Optional[date] without a default value are considered "Required, can be None," meaning they are expected to be present in the data, albeit they can be None. However, for fields to truly be optional—where they can be omitted from the data entirely without triggering validation errors—they should be defined with a default value, e.g., ex_date: Optional[date] = None. This change marks the fields as "Not required, can be None, and is None by default," allowing them to accommodate missing fields in the API response.

It appears that the CorporateActionAnnouncement class definition needs updating to reflect this understanding, specifically by assigning default values to optional fields. The test case appears to include all of the fields so wouldn't catch this scenario.

hiohiohio commented 8 months ago

@DougForrest thank you for heads-up. I will handle this issue.

hiohiohio commented 3 months ago

prev fix PR: https://github.com/alpacahq/alpaca-py/pull/426

alpacahq / alpaca-py