Wrong type error when passing Integer

mkondapalli commented 1 year ago

Version 1.0.0

This is happening for various API calls. When passing the int value to the call it still says string is passed.

fastly.exceptions.ApiTypeError: Invalid type for variable 'version'. Required value type is int and passed type was str at ['received_data']['version']

This is the API call where version_is is int, I tried type casting too but still the same problem persists.

I observed this in other calls too.

api_response = api_instance.create_snippet(service_id, 44 , name=snippet_name, dynamic=dynamic, type=type, content=content, priority=priority)

Traceback (most recent call last): File "/Users/kondapal/fastly_py/main.py", line 47, in vcl.createVCLSnippet(configuration, service_id, int(cloned_service_version), acl_name, vcl_snippet) File "/Users/kondapal/fastly_py/vcl.py", line 43, in createVCLSnippet api_response = api_instance.create_snippet(service_id, version_id , name=snippet_name, dynamic=dynamic, type=type, content=content, priority=priority) File "/usr/local/lib/python3.10/site-packages/fastly/api/snippet_api.py", line 573, in create_snippet return self.create_snippet_endpoint.call_with_http_info(kwargs) File "/usr/local/lib/python3.10/site-packages/fastly/api_client.py", line 862, in call_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.10/site-packages/fastly/api_client.py", line 421, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.10/site-packages/fastly/api_client.py", line 232, in __call_api return_data = self.deserialize( File "/usr/local/lib/python3.10/site-packages/fastly/api_client.py", line 337, in deserialize deserialized_data = validate_and_convert_types( File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1569, in validate_and_convert_types converted_instance = attempt_convert_item( File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1462, in attempt_convert_item raise conversion_exc File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1453, in attempt_convert_item return deserialize_model(input_value, valid_class, File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1373, in deserialize_model return model_class._new_from_openapi_data(kw_args) File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 45, in wrapped_init return fn(_self, *args, kwargs) File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 369, in _new_from_openapi_data return cls._from_openapi_data(*args, *kwargs) File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 45, in wrapped_init return fn(_self, args, kwargs) File "/usr/local/lib/python3.10/site-packages/fastly/model/snippet_response.py", line 233, in _from_openapi_data composed_info = validate_get_composed_info( File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1992, in validate_get_composed_info allof_instances = get_allof_instances(self, model_args, constant_args) File "/usr/local/lib/python3.10/site-packages/fastly/model_utils.py", line 1789, in get_allof_instances raise ApiValueError( fastly.exceptions.ApiValueError: Invalid inputs given to generate an instance of 'ServiceIdAndVersion'. The input data was invalid for the allOf schema 'ServiceIdAndVersion' in the composed schema 'SnippetResponse'. Error=Invalid type for variable 'version'. Required value type is int and passed type was str at ['received_data']['version']

Integralist commented 1 year ago

Thanks @mkondapalli for opening this issue.

I've created an internal ticket to track this issue. We'll report back once we have an update.

mkondapalli commented 1 year ago

Thanks @Integralist any update on the issue? It would be great if you can give us ETA as we have some deliverables and we relied heavily on this library.

Integralist commented 1 year ago

Hi @mkondapalli I've asked one of our team to carry out some investigative work tomorrow. Hopefully we'll have some more feedback after they've had a chance to dig into this error.

mkondapalli commented 1 year ago

@Integralist Can you please let me know if you have any update on this ?

Integralist commented 1 year ago

Hi @mkondapalli

Thanks for your patience, I'm now actively investigating this bug and am trying to reproduce it.

You mentioned you saw this issue with various API calls, do you happen to have a list of those? I see in your issue description you reference one of them, create_snippet() but from my current testing I've not being able to reproduce the error.

Here is the example I created for testing, which lists all backends for a specified service and service version (where version_id is an integer:

import os
from pprint import pprint

import fastly
from fastly.api import backend_api

configuration = fastly.Configuration()
configuration.api_key['token'] = os.getenv('FASTLY_API_TOKEN')

with fastly.ApiClient(configuration) as api_client:
    api_instance = backend_api.BackendApi(api_client)
    service_id = "<REDACTED>"
    version_id = 1

    try:
        api_response = api_instance.list_backends(service_id, version_id)
        pprint(api_response)
    except fastly.ApiException as e:
        print("Exception when calling BackendApi->list_backends: %s\n" % e)

I was able to replicate with create_snippet so I'm going to dig into that now...

import os
from pprint import pprint

import fastly
from fastly.api import snippet_api

configuration = fastly.Configuration()
configuration.api_key['token'] = os.getenv('FASTLY_API_TOKEN')

with fastly.ApiClient(configuration) as api_client:
    api_instance = snippet_api.SnippetApi(api_client)
    service_id = "<REDACTED>"
    version_id = 1
    name = "test-snippet"
    dynamic = 0
    type = "init"
    content = "content_example"
    priority = 10

    try:
        api_response = api_instance.create_snippet(
            service_id, version_id, name=name, dynamic=dynamic, type=type,
            content=content, priority=priority
        )
        pprint(api_response)
    except fastly.ApiException as e:
        print("Exception when calling SnippetApi->create_snippet: %s\n" % e)

What's interesting is that when I check my service, I can see the snippet was actually created successfully regardless of the API client error. If I try and run my above script again I'll see the API returns a 409 Conflict as there is a duplicate snippet.

Inspecting (in fastly/api_client.py) I can see on like 216: https://github.com/fastly/fastly-py/blob/main/fastly/api_client.py#L216

That response_data.data contains:

b'{"name":"test-snippet","dynamic":0,"type":"init","content":"content_example","priority":"10","service_id":"<REDACTED>","version":"1","id":"<REDACTED>","updated_at":"2023-03-16T17:38:22Z","deleted_at":null,"created_at":"2023-03-16T17:38:22Z"}'

...notice the version_id field has 1 as a string. So I'm presuming this is causing issues.

Integralist commented 1 year ago

UPDATE: This looks to be an issue either with our OpenAPI specification or with the API implementation itself.

Our OpenAPI specification states that version_id should be returned as an integer but I can see from the API response (and I tested this manually via the following curl to the API) that it returns it as a string...

curl -i -X POST "https://api.fastly.com/service/<REDACTED>/version/1/snippet" -H "Fastly-Key: $FASTLY_API_TOKEN" -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -d "name=test-snippet-via-curl-1&dynamic=1&type=recv&content=set+req.enable_segmented_caching+%3D+true%3B"
HTTP/2 200
status: 200 OK
cache-control: no-store
content-type: application/json
fastly-ratelimit-remaining: 9993
fastly-ratelimit-reset: 1678989600
accept-ranges: bytes
via: 1.1 varnish, 1.1 varnish
date: Thu, 16 Mar 2023 17:48:52 GMT
x-served-by: cache-control-cp-aws-us-east-2-prod-1-CONTROL-AWS-UE2, cache-lhr7380-LHR
x-cache: MISS, MISS
x-cache-hits: 0, 0
x-timer: S1678988932.832357,VS0,VE239
vary: Accept-Encoding
strict-transport-security: max-age=31536000
content-length: 263

{"name":"test-snippet-via-curl-1","dynamic":1,"type":"recv","content":null,"service_id":"<REDACTED>","version":"1","created_at":"2023-03-16T17:48:51Z","priority":100,"updated_at":"2023-03-16T17:48:51Z","id":"<REDACTED>","deleted_at":null}%

Fundamentally this function in the API client is called and attempts to validate the API response, which causes the fastly.exceptions.ApiValueError exception to be raised.

Integralist commented 1 year ago

@mkondapalli following on from what I discovered yesterday: I've been discussing internally with the relevant team and they're currently identifying the appropriate solution. I don't have an ETA on when it will be resolved unfortunately but I'll update you once I know more.

mkondapalli commented 1 year ago

@Integralist no worries. Please keep us posted when you release a newer version with the fix.

mkondapalli commented 1 year ago

@Integralist any update on this?

Integralist commented 1 year ago

👋🏻 Hi @mkondapalli

Thanks for your patience.

I can confirm that we are going to be updating our internal OpenAPI schemas to more accurately reflect the 'types' returned by the Fastly API, and once that is done I'll be able to regenerate all of our API clients (which will include fastly-py). This should then resolve the issue you've reported here.

Unfortunately, I can't give an ETA on when the work will be completed as we need to go through an internal audit process to identify all the affected endpoints but be assured that, as of next week, this will be actively worked on as a priority.

Any other questions, please just let me know.

Thanks!

Integralist commented 1 year ago

👋🏻

We've published 2.1.1 of fastly-py which resolves prior issues with creating snippets (code example below)...

import os
from pprint import pprint

import fastly
from fastly.api import snippet_api

configuration = fastly.Configuration()
configuration.api_key["token"] = os.getenv("FASTLY_API_TOKEN")

with fastly.ApiClient(configuration) as api_client:
    api_instance = snippet_api.SnippetApi(api_client)
    content = "content_example"
    dynamic = 0
    name = "test-snippet-1"
    priority = "10"  # this is now a string and not an int
    service_id = "<REDACTED>"
    snippet_type = "init"
    version_id = 99

    try:
        api_response = api_instance.create_snippet(
            service_id,
            version_id,
            name=name,
            dynamic=dynamic,
            type=snippet_type,
            content=content,
            priority=priority,
        )
        pprint(api_response)
    except fastly.ApiException as e:
        print("Exception when calling SnippetApi->create_snippet: %s\n" % e)

In the snippet response both priority and version fields are now strings instead of integers.

I'm going to close this issue for now, but please continue the discussion if you're still having issues.

Thanks!

Markb1337 commented 1 year ago

@Integralist I've already reached out about this via support@fastly.com, but better safe than sorry: This issue still exists. I've encountered it in several places while trying to use the fastly-py library. Just tested with the newest version (3.0.2), and got the following error when creating a cache setting using 'create_cache_settings' from 'cache_settings_api.CacheSettingsApi':

fastly.exceptions.ApiValueError: Invalid inputs given to generate an instance of 'CacheSetting'. The input data was invalid for the allOf schema 'CacheSetting' in the composed schema 'CacheSettingResponse'. Error=Invalid type for variable 'ttl'. Required value type is int and passed type was str at ['received_data']['ttl']

When checking the API response for creating cache settings using curl, I noticed that the following values return incorrect data types: 'ttl', 'stale_ttl' and 'version'.

Integralist commented 1 year ago

Hi @Markb1337

Thanks for raising this issue. There was indeed a problem with our OpenAPI schemas that had these fields documented as integers when in fact the API returns them as a string. I've corrected this and hope to get a new release of the fastly-py API client published next week.

Integralist commented 1 year ago

👋🏻

https://github.com/fastly/fastly-py/releases/tag/release%2Fv4.0.0

https://github.com/fastly/fastly-py/blob/main/CHANGELOG.md#v400-2023-07-31

Markb1337 commented 1 year ago

Hi,

Thanks for the release and your hard work!

Sadly I'm still encountering issues with the new version.

There now seems to be a new bug in the API/Python library; Parameters which are optional when creating new objects are of a specific type (i.e. 'str'), but as they are optional parameters, it could happen that objects are created without any value for this parameter. When the API returns the object (i.e. after creation, or when listing certain objects via the API), the value will be of type 'Nonetype', which is not allowed as the API definition says it should be 'str'. I guess optional parameters should also be allowed to be of type ‘Nonetype’.

A practical example; As I was testing the new version, the following happened:

I created a cachesettings object with the Python library;
I did not provide a stale_ttl parameter, as this is optional according to the documentation (https://github.com/fastly/fastly-py/blob/main/docs/CacheSetting.md);
Creating the cachesettings object is successful, so it appears the stale_ttl parameter is indeed optional. However, the API response contains a Nonetype value for the stale_ttl as it did not receive any value during creation;
The python library throws an error as ['received_data']['stale_ttl'] is of type Nonetype where the library expected a string, see the error below:

_fastly.exceptions.ApiTypeError: Invalid type for variable ‘stale_ttl’. Required value type is str and passed type was NoneType at [‘received_data’][‘stalettl’]

I also reported another breaking bug earlier (via the support desk) which does not seem to be fixed in version 4.0.0, I'll create a separate issue for that bug in a minute.

Thanks for your work, much appreciated! Mark

Integralist commented 1 year ago

👋🏻 Hi @Markb1337

My sincerest apologies for the issues you have been encountering. The code-generation process has been problematic and internally we're very much aware that a lot of improvement is needed and we're actively scheduling time to work on it.

We'll dig into the above issue (and the separate one you reported) soon and will report back.

We really appreciate your patience and understanding.

Integralist commented 1 year ago

Hi @Markb1337

Just a quick update to let you know I've made a bunch of tweaks to our internal OpenAPI schemas to address instances where we were missing nullable: true being set on certain fields (this includes the stale_ttl field).

I'm waiting for further feedback from the relevant Fastly team and then hopefully I'll be able to get those changes merged and a new fastly-py client regenerated from the updated schemas.

Markb1337 commented 1 year ago

Hi @Integralist

Thanks for the update! Good to hear things are coming along.

Appreciate the hard work and transparency,

Mark

Integralist commented 1 year ago

Hello, just another quick update.

I've been on PTO for the last couple of weeks and, since my return, it turns out there have been some unexpected blockers raised with regards to my work tweaking the internal Fastly OpenAPI schemas.

Effectively the work required to fix the issue with the OpenAPI schemas has become a lot larger and I'm now reaching out to various Fastly teams related to this.

Rest assured this is something that's being actively worked on and I'll post updates as I have them.

Thanks for your patience.

Integralist commented 1 year ago

To avoid too much noise in this closed issue I've now raised:
https://github.com/fastly/fastly-py/issues/95 for customers to track updates.

fastly / fastly-py

Wrong type error when passing Integer #88