Providing details of API calls

pmonno commented 2 years ago

Summary API usage is considerably higher than expected - we'd like to get a breakdown of each endpoint call to better understand what is the cause for this.

Detail and or supporting links and screenshots Our website should only need to pull in Properties, PropertyImages, and Documents (on request) - but is pulling in everything.

We would like to understand if something in the code is triggering all endpoint calls; however, the Developer portal only shows numbers and calls, which we hasn't been able to help our investigation.

AshDeeming commented 2 years ago

Hi @pmonno Please could you confirm the app name or ID.

pmonno commented 2 years ago

Hi @AshDeeming The app name is "Carter Jonas Website Integration" ID "3d511e0b-3c90-4ccc-99c7-d9d1f034dcb3"

plittlewood-rpt commented 2 years ago

HI @pmonno

I've had a look at logs from the last 8 hours and make the following observations:

You are downloading physical file assets multiple times. For instance you called /documents/BAT22005962/download 4 times since 4am this morning, once at 04:52am, then at 07:15, 07:29 and 07:30 respectively. The latter three calls seem odd given their close proximity to each other
You are making seemingly excessive requests to the properties endpoint if your aim is to only fetch changes since your last import. It looks like you could be processing the entire dataset. There's a suggested architecture for website feeds here

Without understanding the expectations of your import schedule it's difficult to advise on where you might be going wrong, but the logs suggest you're processing the same data over and over again, which is likely unnecessary.

I hope this helps Thanks

plittlewood-rpt commented 2 years ago

Hi @pmonno please let me know if you need any further assistance with this. Thanks

pmonno commented 2 years ago

Hi @plittlewood-rpt is would it be possible to provide details of the calls which have been recorded?

Our devs would like to understand if a single API call to properties is spawning other API calls for other property related data when it is called with embed= parameters?

himynameistim commented 2 years ago

Hi @plittlewood-rpt

Thanks for your feedback.

I've reviewed the code in our solution and we can not see any calls being made to the documents, offices, propertyimages, tenancies and vendors apis. However the analytics shows between 20,000 and 50,000 calls being made to these APIs.

What we are doing though is setting the embed parameter on the properties API to include images, offices, tenancies, landloards, documents, negotiator and vendor.

When embeding is used does one call to the property API count as calls to all the others when there is related data?

plittlewood-rpt commented 2 years ago

Yes that’s right. The embed mechanism is a helper mechanism that just saves you making multiple calls yourself to fetch related data. There will not be a one to one relationship either, for example if a property has a lot of images and you call properties/?embed=propertyImages it will page through the images data as required to fetch all the associated content.

Please see https://foundations-documentation.reapit.cloud/api/api-documentation#embedding-data

From: Tim @.> Sent: Tuesday, October 11, 2022 5:36:10 PM To: reapit/foundations @.> Cc: Pete Littlewood @.>; Mention @.> Subject: [EXTERNAL] Re: [reapit/foundations] Providing details of API calls (Issue #7839)

External Message - Please be cautious when opening links or attachments in email

Hi @plittlewood-rpthttps://github.com/plittlewood-rpt

Thanks for your feedback.

I've reviewed the code in our solution and we can not see any calls being made to the documents, offices, propertyimages, tenancies and vendors apis. However the analytics shows between 20,000 and 50,000 calls being made to these APIs.

What we are doing though is setting the embed parameter on the properties API to include images, offices, tenancies, landloards, documents, negotiator and vendor.

When embeding is used does one call to the property API count as calls to all the others when there is related data?

— Reply to this email directly, view it on GitHubhttps://github.com/reapit/foundations/issues/7839#issuecomment-1274975339, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHK7GIY73BWS6T7ERSKAYZLWCWJPVANCNFSM6AAAAAAQ4UWPZE. You are receiving this because you were mentioned.Message ID: @.***>

himynameistim commented 2 years ago

@plittlewood-rpt can you confirm if setting ModifiedFrom parameter takes into account modifications to data that is embedded.

e.g. If a Tenancy related to a Property was updated today and we called the Property API embed tenancy and set the ModifiedFrom flag to yesterday, would the Property and the updated Tenancy be returned or would it only get returned if the Property was updated too?

plittlewood-rpt commented 2 years ago

Hi @himynameistim - the modifiedFrom parameter does not get cascaded down to embed calls no. This was a conscious decision when the system was designed as it's likely some developers would want that behaviour, but others would not. That said, it's not beyond the realms of possibility and could be looked at again if needed

plittlewood-rpt commented 2 years ago

Hi @himynameistim do you need anything else from us on this at this point? There is some documentation here around website development, with a specific point 2 that mentions not to embed images after your initial database seed for this reason. If you want us to review allowing the modifiedFrom parameter to cascade to embedded services, please open another ticket. Many thanks

reapit / foundations

Providing details of API calls #7839