search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.47k stars 3.99k forks source link

[source-shopify] Missing Collections Stream #43308

Open karolur opened 1 month ago

karolur commented 1 month ago

Connector Name

source-shopify

Connector Version

v2.4.16

What step the error happened?

Other

Relevant information

Description

The source-shopify connector documentation states that a Collections stream is available (https://docs.airbyte.com/integrations/sources/shopify#supported-streams). However, this stream is not present in the current version of the connector. I used to be able to access this stream back in February of this year (unfortunately, I do not recall the exact version of the connector back then). Currently, I can only find the following streams:

smart_collections
custom_collections
metafield_collections

Please advise on how we can resolve this issue or if there is an alternative approach to access the Collections data.

Thank you

Relevant log output

No response

Contribute

marcosmarxm commented 1 month ago

The Airbyte's Shopify sandbox still has access to this stream @karolur. Maybe it is a lack of permissions in your account? I left it open to see if other users are having the same problem.

karolur commented 1 month ago

@marcosmarxm hey, thank you for replying. I'm using the OSS version, I'm not aware of any permissions needed to access certain streams in a connector, could you please elaborate ?

If you refer to the scopes needed to access the Collection streams, we have the scope 'read_products' needed (https://shopify.dev/docs/api/admin-graphql/2024-04/objects/Collection)

I just checked and the only scope we don't have among the listed ones in the airbyte docs is 'read_publications' but we've never had it and we were able to see the Collections there back in February. https://docs.airbyte.com/integrations/sources/shopify#custom-app-scopes

Can you think of something else that might be blocking us from seeing this stream in the Shopify connector? out of all the streams listed in the docs the only one we can't see is Collections

altumblack commented 1 month ago

What version of OSS are you using? I'm able to see this in my OSS 0.63.12? The issue I'm having is that Airbyte doesn't show what Products belong to what Collection.

karolur commented 1 month ago

What version of OSS are you using? I'm able to see this in my OSS 0.63.12? The issue I'm having is that Airbyte doesn't show what Products belong to what Collection.

I have Airbyte deployed in GKE using Helm chart 0.248.5 and Airbyte version 0.63.4 To relate the Products to the Collections you need the Collects table https://shopify.dev/docs/api/admin-rest/2024-04/resources/collect#top