Does `tb-exporter` still work?

[Kabouik]

Thanks for this tool!

I would need it export raw data from devices listed on a Thingsboard CE but I am getting errors. I'm getting multiple different errors because I am still experimenting with variations of the dashboard URL, device names, etc., so I may very well be just be misusing the CLI, but will post relevant errors if/when I'm sure I'm using the tool appropriately. However before going too deep in the rabbit hole and figuring out if the errors are due to misuse, to my server, or to dependencies on my client machine, I would like to know if tb-exporter has been used recently and been confirmed to work with the latest API of Thingsboard.

Thanks!

[abalmos]

Hi @Kabouik. We wrote this to move away from Thingsboard, so we no longer have an instance to test with. If you are interested in maintaining it, I would be happy to give you access.

[Kabouik]

Hi @abalmos, thanks for the prompt reply and for the offer! This may be a bit early as I am just beginning with Thingsboard (and may not know the limitations that made you move from it) and am not sure how to fix the script yet (if it is indeed no longer working), but I appreciate the proposal.

I just have a number of devices currently sending data to Thingsboard and it's working great so far, but I need to get csv exports of the raw data for each device for further analyses, and it seems this feature would require Thingsboard Pro without your tool, we cannot afford it (nor do we want to support a paywall strategy for such a core feature).

I will try again the script later and see if I can understand the errors.

[abalmos]

We moved away for pretty much that reason. ThingsBoard is neat, but you quickly feel trapped unless you have a pro license. The cost was prohibitive for use. We now use Benthos to drop our event data into a TimescaleDB database and then plot it with Grafana.

[Kabouik]

Frustrating. I wonder now if picking TB was a good idea for us either. We were offered that solution and it looked convenient but we didn't check the limitations at that time. Grafana would be nice indeed, and it has a dark mode at least. However, we need to be able to send parameters to devices, not sure if Grafana can do this.

Anyway. The script still works after a minor adjustment: L50 needs to be replaced with p = p._append(t)

The reason I saw many other errors before that simple one was because I was trying to use it with real devices on a dashboard I am not administrating or even registered as a tenant on. Consequently, I couldn't see the devices IDs, which seem to be accessible only from the tenant profile. The devices I was using for --device were either the device names, device labels, or some hash that resembled a device ID but was likely the dashboard ID. I tried the script on a dummy device on my own dashboard where I am admin and have a tenant account, and it worked fine:

for i in 0 1 2 3 4 5 6 7 8 9
do curl -v -X POST https://<TB_host>:443/api/v1/<access token>/telemetry \
  --header Content-Type:application/json \
  --data "{light:$i,temperature:$i,rh:$i,ntc0:$i,ntc1:$i,ntc2:$i,ntc3:$i}"
done

$ python3 fetch.py https://<TB_host> -u <user> -p <password> -d <deviceid>
$ cat data-*.csv
ts,temperature,ntc0,ntc1,ntc2,ntc3,rh,light
1717801539778,9,9,9,9,9,9,9
1717801539652,8,8,8,8,8,8,8
1717801539502,7,7,7,7,7,7,7
1717801539326,6,6,6,6,6,6,6
1717801539125,5,5,5,5,5,5,5
1717801538924,4,4,4,4,4,4,4
1717801538749,3,3,3,3,3,3,3
1717801538598,2,2,2,2,2,2,2
1717801538447,1,1,1,1,1,1,1
1717801538297,0,0,0,0,0,0,0

Now I wonder, would it be hard to alter the script so that it can accept device names too (they are unique too), or labels (I guess they may not be unique but that could be handy to use them as tags/keywords and fetch data from all matching devices at once), or from all devices known to the user when no device is specified?

Also, do I really need a tenant authentication to download the data? I needed it to find the devices ID, but if a customer is provided a list of ID by their tenant, then I wonder if they would be able to fetch the data.

[Kabouik]

My understanding is the script uses the REST API with /api/plugins/telemetry/{entityType}/{entityId}/values/timeseries{?keys,useStrictDataTypes} which only seems to take deviceid when the entity is set to DEVICE, and I see no alternative with device names or customer permissions.

I guess one way to allow device names and categorries/tags instead of deviceid only would be to first have the user write a csv containing the correspondence between deviceid, devicename and tags, and then add two options in fetch.py to provide either a name or a tag instead of an ID, and if so, find the ID correspondence in that file to adjust the GET request accordingly.

[Kabouik]

Hum, weird hack. I had to inspect the page to remove the spam image. Let me clarify that this user has nothing to do with me, but the coincidence is kinda weird since there was no open issue here before I posted earlier today. :/ Concerning.

[Kabouik]

My bad, it may actually use getTimeseries which is also available to customers, but customers still have no way to get the device ID from their dashboard, so the tenant will have to provide them. I'll try.

In any case, I think it'd be interesting to develop a way to use devicenames and/or tags from an external csv file too. I'll look into this when I have devices connected to my own Thingsboard server.

Would you happen to know if there is a limitation to the number of REST requests with the Community Edition?

[abalmos]

I don't believe there are limitations on the number of requests in a self-hosted community edition. Certainly, the tool would be more useful if it could work against device names and tags. I recall that was going to be somewhat difficult, so we just managed that by hand for the few devices that we had.

[Kabouik]

Yes, I did check the limitations of Thingsboard CE and there are none on the API, the admin is just free to set them if need by. Sorry I forgot to report it here.

Since it works for me with the minor change above, and since I have close to 100 devices, I will likely give a go at a way to use device names (tags might be more difficult and less useful for me at least on the short term), provided that the user has a pre-made table with IDs and names. Can't promise I'll manage, as I'm not particularly good with Python, but I'll let you know. I cannot do it right now though, but as far as I understand the project is dormant currently anyway.

On 2024-06-20 07:43 Andrew Balmos @.***> wrote:

I don't believe there are limitations on the number of requests in a self-hosted community edition. Certainly, the tool would be more useful if it could work against device names and tags. I recall that was going to be somewhat difficult, so we just managed that by hand for the few devices that we had.

--
Reply to this email directly or view it on GitHub: https://github.com/oats-center/tb-exporter/issues/1#issuecomment-21798 60487 You are receiving this because you were mentioned.

Message ID: @.***>