CloudQuery Source Plugin?

[yevgenypats]

Hi Team, hopefully this is right place to ask, if not, I'd appreciate if you can direct me.

I'm the founder of cloudquery.io, a high performance open source ELT framework.

Our users are interested in a Simple Analytics plugin, but as we cannot maintain all the plugins ourselves, I was curious if this would be an interesting collaboration, where we would help implement an initial source plugin, and you will help maintain it.

This will give your users the ability to sync Simple Analytics data to any of their datalakes/data-warehouses/databases easily using any of the growing list of CQ destination plugins.

Best, Yevgeny

[adriaandotcom]

Hi @yevgenypats, we are for sure open to that. Maintaining it is usually adding a few extra fields. Happy to do that.

How do you want to connect? I think it's best to get the raw values as streaming data, right?

See https://docs.simpleanalytics.com/api/export-data-points

[yevgenypats]

Hi @adriaanvanrossum , That's awesome! Yeah I think the export-data-points API would work.

What we will do then is create a first version and test it under cloudquery organization and then once it is working and you have a quick chance to review we can transfer the repo to simpleanalytics.

The plugin will be linked from our main docs under community https://www.cloudquery.io/docs/plugins/sources/overview.

If that sounds good we can start working on that!

[adriaandotcom]

Perfect, if you need an account to test, or test data, let me know.

[yevgenypats]

Perfect, if you need an account to test, or test data, let me know.

Awesome. We are using Simple analytics as well so we have a test account :) will keep you posted!

[hermanschaaf]

Hi @adriaanvanrossum 👋 I'll be working on bootstrapping the plugin, could you please share test data for the export API (or access to a test account), or anything else you may have? Sounds useful, and I'll probably include it in the test suite :)

[adriaandotcom]

You can use these credentials:

• saasforcovid.com
• sa_user_id_77969473-8121-4ef4-882b-2bda8acc7fc3
• sa_api_key_xwPSzcqDIjb4xNZVM76WYMb3LNCbstdkmttT

[hermanschaaf]

Hi again @adriaanvanrossum, thanks for the test credentials! Are there any metadata fields available for that hostname that we can test against as well?

[adriaandotcom]

I added a few events with metadata, if you need more, just add them here:

sa_event("metadata_event", {
  updated: new Date(),
  boolean: Math.random() > 0.5,
  decimalCount: 123.456,
  string: 'Hi\ntherë'
})

[hermanschaaf]

Hi @adriaanvanrossum, thanks for your help! I'm happy to report that the first version of the Simple Analytics CloudQuery source plugin is now ready, and we are already using it ourselves to export SA data to BigQuery on a daily basis 😃

Here is the repo if you'd like to take a look: https://github.com/cloudquery/cq-source-simple-analytics

And here is a sample repo with a Github Action workflow that runs a sync to BigQuery on a daily basis. This is what we're using ourselves, and it also serves as a handy public reference that we'll keep up-to-date over time: https://github.com/cloudquery/recipes

If you have any questions I'd be happy to answer, or if you'd like to jump on a call for a live demo/Q&A, that would also be great. If you feel like it looks good, we can migrate the repo over to the simpleanalytics org (either via GitHub migration or if you create the repo I can open a PR to it).

[adriaandotcom]

That's awesome! Thanks a lot. We would love to add CloudQuery to our integrations page. Do you have something we can copy? Here is an example with Chartbrew.

[hermanschaaf]

Nice! That would be great. I'd be very happy to write something. In the CloudQuery case there would be many possible destinations, each with slightly different setup steps, mostly around authentication. So maybe it can be a tutorial called "Export to Database or Data Warehouse" under the Integrations section? We can have an example that shows how to export to BigQuery to give the general idea, and link to the CloudQuery docs for all the other databases (Snowflake, Athena, Postgres, MSSQL, etc). This way the Simple Analytics specific parts will be clearly explained with an example, and the destination docs can serve to fill the gaps for all the potential destinations. How does that sound?

[adriaandotcom]

That sounds perfect. If it could be markdown, that would be best for us. And usually the more images, the better :)

[hermanschaaf]

Sounds good, I'll work on something this week :)

[hermanschaaf]

Hi @adriaanvanrossum :wave: Could you open an empty repo at simpleanalytics/cq-source-simple-analytics (or simpleanalytics/cq-source-simpleanalytics without the dash, as you prefer), then we can fork and open a PR to bring in the changes from https://github.com/cloudquery/cq-source-simple-analytics?

[adriaandotcom]

@hermanschaaf here you go: https://github.com/simpleanalytics/cq-source-simpleanalytics/invitations

[hermanschaaf]

@adriaanvanrossum Great! Here is the first draft of the how-to guide if you'd like to take a look: https://github.com/simpleanalytics/cq-source-simpleanalytics/pull/1

It's intended to be listed in the docs as Export to Database or Data Warehouse. It shows two examples: SQLite (simpler) and BigQuery (a little more involved). It's a bit longer than the average guide on the Simple Analytics website, but I think this gives enough context to be allow readers to generalize to any destination, without creating a maintenance burden for you.

Let me know what you think, and if you're happy how we can integrate it into the SA docs!