Device importer seems to need semicolons, not commas

[eggfriedrice]

Summary

Hey

Weird one, I tried to import a CSV of devices, based on the Excel template provided (but trimming down to the required fields). I saved this file using LibreOffice and then munged it in VS Code.

The importer failed with my file, giving "Import process cancelled due to error."

Checking the examples again, I noticed they had semicolon separators. I did a find-and-replace to switch my commas to semicolons and then the same file imported fine.

I think the importer should accept CSVs, because at the moment it looks like it only accepts "SSVs"! Unless there's some weird edge case.

My file looked like this:

id,dev_eui,join_eui,name,frequency_plan_id,lorawan_version,lorawan_phy_version,app_key,band_id eui-xxxxxxxxxxxxxxxx,xxxxxxxxxxxxxxxx,xxxxxxxxxxxxxxxx,New Elsys ELT,EU_863_870_TTN,MAC_V1_0_3,RP001_V1_0_3_REV_A,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,EU_863_870

...

Steps to Reproduce

Upload a CSV

Current Result

CSV not imported

Expected Result

CSV imported!

Relevant Logs

No response

URL

No response

Deployment

The Things Stack Cloud

The Things Stack Version

3.2.0

Client Name and Version

Chrome 103.0.5060.53 (Official Build) (64-bit)

Other Information

No response

Proposed Fix

No response

Contributing

• [ ] I can help by doing more research.
• [ ] I can help by implementing a fix after the proposal above is approved.
• [ ] I can help by testing the fix before it's released.

Code of Conduct

• [X] I agree to follow TTN's Community Code of Conduct.

[nicholaspcr]

With the current implementation of the TTS-CSV importer, the delimiter is hard-coded as a semicolon. Considering that LibreOffice exports csv files using comma as the default delimiter this is something that should be added to the TTS-CSV.

I can work on this, probably best to leave the default delimiter as a comma considering that this is what is used in LibreOffice and Microsoft Excel. While changing the default delimiter the option to overwrite it should be include on the implementation of this as well.

[johanstokking]

@eggfriedrice can you confirm that LibreOffice exports CSV file with default settings with delimiter , and not ;?

Microsoft Excel and Apple Numbers export CSV with ; as separator, so we used that.

[nicholaspcr]

I didn't write on my previous comment but I checked on Linux using LibreOffice and the default delimiter is indeed a comma.

[kschiffer]

Unfortunately this is still an issue. Semicolons do not work properly:

Consider:

dev_eui;join_eui;frequency_plan_id;lorawan_version;lorawan_phy_version;app_key
2C6EADDE47DCE3F1,4E7E4849D859ABCB,EU_863_870_TTN,MAC_V1_0_4,RP002_V1_0_3,B17AAB0B1E428278C2CAE71F6527B0AF
30751BF43AB55A64,39633F47CCF23125,EU_863_870_TTN,MAC_V1_0_4,RP002_V1_0_3,B65D240BB219CB6402D1C3CC4F8757C0
55E233905FC89A4C,5A4BC689178AD1B2,EU_863_870_TTN,MAC_V1_0_4,RP002_V1_0_3,D3CD3B4E8B565206C369E6F67E35295A

Yields:

Converting end device templates…
{
  "grpc_code": 3,
  "http_code": 400,
  "message": "error:pkg/devicetemplates:parse_csv (parse CSV at line 2 column 1: wrong number of fields)",
  "http_status": "Bad Request",
  "details": [
    {
      "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
      "namespace": "pkg/devicetemplates",
      "name": "parse_csv",
      "message_format": "parse CSV at line {line} column {column}: {message}",
      "attributes": {
        "column": 1,
        "line": 2,
        "message": "wrong number of fields",
        "start_line": 2
      },
      "correlation_id": "09b586a0b6e347ed8caa435c7b225627",
      "code": 3
    }
  ]
}

Import process cancelled due to error.

[eggfriedrice]

Hi Kevin, it looks like you have a mix of semicolons in the first line and commas in the following ones. Can you try converting to all commas?

[kschiffer]

Yes, it is a mix. It's kind of an edge case but I think it should still work. The import feature should be maximally forgiving, or it should at least output an error message that is more helpful.

Earlier I wrote that using just semicolons does also not work, but that was not the case actually. As such I will close this issue again.

[eggfriedrice]

Ahh, I see. I think it's reasonable to bomb out as that's not a valid CSV or "semicolon-SV" file.