steilerDev / icloud-photos-sync

One-way sync engine for the iCloud Photos Library into the native file system
https://icps.steiler.dev/
GNU General Public License v3.0
340 stars 9 forks source link

Provided Grafana Dashboard example does not show any data #329

Closed halbtuerke closed 12 months ago

halbtuerke commented 1 year ago

Checklist before opening an issue

Describe the bug

I have enabled the metrics export and configured InfluxDB, telegraf and Grafana. As far as I can tell, all system are up and configured to communicate. Looking into the InfluxDB web UI, I can see some data from the syncing process. After importing the provided Grafana dashboard example JSON, unfortunately the dashboard on my machine looks like this:

CleanShot 2023-09-14 at 14 27 32@2x

I have no experience with Influx, telegraf or Grafana. Does the provided dashboard have any specific prerequisites, which I need to adhere to? I tried looking into the JSON but I can't make heads or tail out of it.

Operating environment

steilerDev commented 1 year ago

Did you specify your data source upon importing the dashboard?

halbtuerke commented 1 year ago

Did you specify your data source upon importing the dashboard?

Yes, I have a InfluxDB data source with the Flux query language, which has access to 3 buckets. When importing the JSON, I have chosen this data source to be used.

steilerDev commented 1 year ago

Are you using an Influx 2 DB? That might be it - I'm still on 1.X (and have created the Dashboard for that) - I'll check this

halbtuerke commented 1 year ago

Yes, I have installed InfluxDB in version 2. I might retry with V1 when I have some spare time to set it up.

steilerDev commented 1 year ago

InfluxDB 2 dropped support for InfluxQL (which has a nice GUI support in Grafana - which is why I stuck with it). So for InfluxDB 2 we'd need to migrate the queries to flux.

If you feel like it you could try doing that and contribute back instead of going back to InfluxDB 1 :)

steilerDev commented 12 months ago

I have looked into this and started migration, until I saw that Flux has actually reached EoL and will not be present in InfluxDB 3. This means there is no point in migrating to Flux - but there is a way to use InfluxQL against InfluxDB 2 - I will parameterize the metric name, as this will be unique to your DBRP mapping

I will add some notes about this to the docs - feel free to share your experience setting this up (as I will probably skip v2 and go straight to v3 once available.

steilerDev commented 12 months ago

Updated the dashboard on the dev branch and added to following documentation (I hope it is clear enough):

When importing the JSON model you need to provide an InfluxDB datasource that supports InfluxQL. InfluxDB 1.X supports this out of the box, InfluxDB 2.X needs to be configured to support InfluxQL.

Additionally you need to specify the measurement name, where the sync metrics are stored. For InfluxDB 1.X this should always be icloud_photos_sync (as this is specified in the Influx Line Protocol written by this tool), InfluxDB 2.X however needs to include the database and retention policy mapping in the measurement query. This could be something like example-db.example-rp.icloud_photos_sync, but depends on the DBRP mapping you previously created.

halbtuerke commented 12 months ago

Wow, awesome! Thank you for your efforts.

I set up the DBRP mapping for the bucket and when using the influx shell, I am able to query it via SQL.

These are the command that I have used:

influx v1 dbrp create \
  --db example-db \
  --rp example-rp \
  --bucket-id 1933412d2ed108d5 \
  --default
influx v1 shell
select * from "example-db"."example-rp".icloud_photos_sync

Unfortunately I cannot create a InfluxDB data source in Grafana, with the InfluxQL query language. No matter what I try, I only get the message: InfluxDB returned error: error reading influxDB after clicking on "Save & test". The same credentials work without any issues with the Flux query language. I'm honestly a little bit baffled by this issue.

Do you have an idea what could be wrong here?

halbtuerke commented 12 months ago

I found the issue. The Authorization header value requires the Token prefix. This is described at https://community.grafana.com/t/integration-with-influxdb-influxql-error-badrequest/42987/11

Now it seems to work! 🥳

CleanShot 2023-09-22 at 23 11 09@2x
steilerDev commented 12 months ago

Great news - and phew...100k assets, let me know how the sync goes, i tried really hard to make it stable, but only got 14k in my library to test with

github-actions[bot] commented 12 months ago

This issue should be resolved with version v1.2.1-beta.1, please confirm.

github-actions[bot] commented 11 months ago

This issue was resolved with version v1.3.0.