search

OS2iot / OS2iot-backend

This repository contains the backend to the project OS2iot.
Mozilla Public License 2.0
10 stars 7 forks source link

Bulk import probably times out. #130

Closed aNorrah closed 2 years ago

aNorrah commented 2 years ago

We've just tested a bulk import of exactly 50 sensors.

We had to import the CSV several times to import all sensors - and technically failed to do so with one file.

The first import managed to import 29 sensors.

At first, we waited. Then we said, let there be light... and slowly a few sensors were added to the application count at a time. First 1, then like 8, then a few refreshes more... etc. The next day we were at 29. Which we probably reached within a few minutes, but since we were unaware of the issue at the time, we kept waiting.

So the next day we added the entire CSV again. Got the following errors on most sensors. Even some that were successfully added, so not sure what they all mean.

SensorX | LoRaWAN | Failed | Id skal være unikt. Det pågældende Id er allerede brug SensorX+1 | LoRaWAN | Failed | write tcp XX.x.0.154:46986->XXX.x.0.56:5432: write: connection reset by peer SensorX+2 | LoRaWAN | Failed | EOF

Unique ID makes perfect sense. We are adding the same sensor several times, so this one should be there. But it wasn't consistent.

About 4 more sensors were added during the 2nd import.

We persisted with the same CSV file, and after 5 imports we had 49/50 sensors on.

The last refused to be added through the bulk import file. No matter how many times we tried.

The error would still read a mixture between the error codes shown above, so just because there were 49 added already, it didn't just read unique ID error for all but one.

We had to manually figure out which was missing, and add it on its own.

aNorrah commented 2 years ago

There is no apparent logic towards the order in which the sensors were added. The last sensor to fail was sensor 26. Sensor 50 was added in the first run through.

ramogens commented 2 years ago

Thanks for thorough reporting @aNorrah. We'll put the bug in the maintenance backlog. I'll just add the detail from speaking to you, that all sensors was LoRaWAN. This means this bug might be connected to the general issue of timeout of calls to the Chirpstack API in the backend.

aNorrah commented 2 years ago

@ramogens That was obviously implied with the "probably times out". Who needs all that information fleshed out. Can't you just all read it with all my background knowledge flawlessly and implicitly understood? wink smiley for implied context

ramogens commented 2 years ago

With today's release of version 1.1.0 this issue should be solved.