Error when importing Fuelly formatted CSV

[ZHoob2004]

I'm unable to import a "Fuelly" CSV and get the following error

2023/02/21 23:38:07 [Recovery] 2023/02/21 - 23:38:07 panic recovered:
No Locale Info Found
/go/pkg/mod/github.com/leekchan/accounting@v1.0.0/unformatnumber.go:21 (0x9cd71a)
/api/service/fuellyImportService.go:66 (0x9d2152)
/api/service/importService.go:101 (0x9d4b05)
/api/controllers/import.go:22 (0x9dc60d)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x9b1cf9)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/recovery.go:99 (0x9b1ce0)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x9ed60b)
/api/main.go:68 (0x9ed5ed)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x9b1cf9)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/recovery.go:99 (0x9b1ce0)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x9b0dd3)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/logger.go:241 (0x9b0d92)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x9a7589)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/gin.go:489 (0x9a756f)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/gin.go:445 (0x9a705b)
/usr/local/go/src/net/http/server.go:2887 (0x6b5b22)
/usr/local/go/src/net/http/server.go:1952 (0x6b0f4c)
/usr/local/go/src/runtime/asm_amd64.s:1371 (0x473e60)

This happens with either of the example exports from the akhilrex repo as well as my hand-formatted example from another app.

Using latest docker image

[ZHoob2004]

After some further debugging, I've identified the problem as user.Currency containing the fully expanded currency format "US dollars (USD" when accounting.UnformatNumber expects the format to be "USD".

I'm tempted to simply use a regex to extract the substring between the parentheses, but that feels inelegant

Regex form here

var rgx = regexp.MustCompile(`\((.*?)\)`)
totalCostStr := accounting.UnformatNumber(record[9], 3, rgx.FindStringSubmatch(user.Currency)[1])

Feels messy but it works (for USD, the one case I tested), and only runs when importing so very rarely.

[alienp4nda]

The currency model actually contains a prop for the currency code. I'm going to see if I can get the code from the model using the currency value set for the user.

However, once you get past the locale error it seems the csv format may have changed.

@ZHoob2004 would you mind providing the first line on your csv export (headings)? I only have records in fuelly as recent as Oct 2021.

https://github.com/AlfHou/hammond/blob/f07922763ba807bb3c3e70e9bfccdc988dea9782/server/models/currency.go#L12-L21

[alienp4nda]

@ZHoob2004 does your csv contain expenses along with fillups, or just fillups?

[ZHoob2004]

I tried both with and without an expense line but wasn't getting past the locale issue.

I had the same locale problem when importing any of the examples from the akhilrex/hammond repo and those contain expenses

[alienp4nda]

Good to know, once the #49 pull request is merged you should at least be able to import fillups.

[alienp4nda]

@ZHoob2004 you should now be able to import using the generic import.

[ZHoob2004]

Thank you very much. I already managed to import mine with my hack a while ago but hopefully this will help others with non-fuelly data sets to import (since I had to reformat mine by hand)

[aspapunktse]

Hi, just built a new docker from latest image but generic import option is not an option, only Drivvo.

Anyone able to share an Drivvo-exported CSV for reference since it is a paid option to export :( and it doesn´t like my Fuelly either.

[AlfHou]

Sorry, didn't catch the conversations here before now. I forgot to make a release after merging @alienp4nda's code. I created a new release now. It should be avaliable on docker in a couple of hours. Docker build failed, will fix tomorrow

[alienp4nda]

Once PR #88 is merged and a new docker image is built I believe this issue can be closed.

If possible, could either @aspapunktse or @ZHoob2004 test out the new image once it's built?

[ZHoob2004]

I no longer have the data I used to test with, and it wasn't "true" fuelly data anyway, since I exported it from another app and reformatted it. There are 2 sample data sets in this issue that can be used to verify. These did not work when I last tested. https://github.com/akhilrex/hammond/issues/11

[alienp4nda]

No worries, and thanks for the data reference. I'll grab those and test those.

[alienp4nda]

There still seems to be issues with with the generic imports when there is only a value when the tank is filled up but no value when the tank wasn't filled up. The generic import also doesn't take into account fillups and services mixed in the same file.

Adding the csv files from the issue by @ZHoob2004 to this issue. Fuelly.Export.2.csv Fuelly.Export.csv

[qtang]

I am running the latest docker build and experiencing the same issue.

Adding my Fuelly Export so you have a current file to work with: FuellyExport.csv

2024/01/20 19:19:07 /api/service/userService.go:29
[0.262ms] [rows:2] SELECT * FROM `user_vehicles` WHERE `user_vehicles`.`user_id` = "bd01996d-f215-41ef-be9d-d895d422e4ac"

2024/01/20 19:19:07 /api/service/userService.go:29
[0.380ms] [rows:1] SELECT * FROM `vehicles` WHERE `vehicles`.`id` IN ("7f10f7b1-b1ce-41f0-83c3-cf3225391d86","ce534264-b7c3-4a14-93db-621be1b69e93")

2024/01/20 19:19:07 /api/service/userService.go:29
[1.030ms] [rows:1] SELECT * FROM `users` WHERE `ID` = "bd01996d-f215-41ef-be9d-d895d422e4ac" ORDER BY `users`.`id` LIMIT 1

2024/01/20 19:19:07 [Recovery] 2024/01/20 - 19:19:07 panic recovered:
No Locale Info Found
/go/pkg/mod/github.com/leekchan/accounting@v1.0.0/unformatnumber.go:21 (0x970e9c)
/api/service/fuellyImportService.go:67 (0x975025)
/api/service/importService.go:102 (0x977344)
/api/controllers/import.go:25 (0x97d3f8)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x953aa1)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/recovery.go:99 (0x953a8c)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x98ac63)
/api/main.go:69 (0x98ac4a)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x953aa1)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/recovery.go:99 (0x953a8c)
[GIN] 2024/01/20 - 19:19:07 | 500 |    4.417313ms |       10.2.1.26 | POST     "/api/import/fuelly"
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x952c06)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/logger.go:241 (0x952be9)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/context.go:165 (0x951cbd)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/gin.go:489 (0x95196b)
/go/pkg/mod/github.com/gin-gonic/gin@v1.7.1/gin.go:445 (0x9514a4)
/usr/local/go/src/net/http/server.go:2936 (0x6bdb75)
/usr/local/go/src/net/http/server.go:1995 (0x6b9091)
/usr/local/go/src/runtime/asm_amd64.s:1598 (0x46f120)

[alienp4nda]

The "No Locale Info Found" error suggests that you're running an older build. Please verify you're running v0.0.24.

[qtang]

I tried running both alfhou/hammond:latest(docker hub) and ghcr.io/alfhou/hammond:latest(github) images. They both are showing the same Digest (sha256:c85dc0293aa10be15280277bade9cce808be572d0d4c5eac62acd87dec30479d).

Both images show "2022.07.06" in the More Info section of the Settings page. Should that be showing "v0.0.24" instead?

docker pull alfhou/hammond:latest
latest: Pulling from alfhou/hammond
Digest: sha256:c85dc0293aa10be15280277bade9cce808be572d0d4c5eac62acd87dec30479d
Status: Image is up to date for alfhou/hammond:latest
docker.io/alfhou/hammond:latest

docker pull ghcr.io/alfhou/hammond:latest
latest: Pulling from alfhou/hammond
Digest: sha256:c85dc0293aa10be15280277bade9cce808be572d0d4c5eac62acd87dec30479d
Status: Downloaded newer image for ghcr.io/alfhou/hammond:latest
ghcr.io/alfhou/hammond:latest

[alienp4nda]

Verified this is an issue, this will take me a bit to troubleshoot.

[alienp4nda]

There are two related issues causing this.

The currency is saved as "Currency Name (CODE)". The settings page does this as does the initialize page on a brand new instance. The backend is expecting only the currency code. I have a fix but this will require everyone who has an existing instance to go to the settings page and select their currency again then save the settings.

[qtang]

It's up to you which way to go, but it looks like the issue is only affecting me at the moment. If you could share a workaround, I could import my data and not inconvenience everybody else.

There are two related issues causing this.

The currency is saved as "Currency Name (CODE)". The settings page does this as does the initialize page on a brand new instance. The backend is expecting only the currency code. I have a fix but this will require everyone who has an existing instance to go to the settings page and select their currency again then save the settings.

[alienp4nda]

It will not just affect you, currently any backend code that needs to use the currency and is expecting the currency code will be affected. I'll just need to figure out a way to notify users.