Bulk import only collects 4/10 tracks.

[philderbeast]

Bulk import of tracks for 2019 Dalmatian XC Paragliding Open - Task 1 says it collected 4/10 tracks when there were actually 8 tracks available.

> tree "output/task 1"
output/task\ 1
├── Igor_Eržen.igc
├── Jerko_Odžak.igc
├── Mitja_Jug.igc
├── Petros_Miskos.igc
├── Rafal_Cypcar.igc
├── Sandi_Krž.igc
├── Slava_Jeliž.igc
└── Stjepan_Zjacic.IGC

[kuaka]

What was the output of the processing log?

[philderbeast]

What was the output of the processing log?

Where do I find this?

[kuaka]

If you are running the 'prod' docker compose file it's the window that opens when processing igc files

[philderbeast]

Here's the section of the log when I'm trying for a 2nd time to bulk import.

We have 6 pilots to find tracks for, and 8 tracks

Sorry but the original bulk import is missing, the buffer doesn't go back that far:

flask_dev_local    | [FLASK] extracting <FileStorage: 'Task 1.zip' ('application/zip')> in dir: /tmp/tmp6autvfbu
flask_dev_local    | [FLASK] Directory: /tmp/tmp6autvfbu
flask_dev_local    | [FLASK]
flask_dev_local    | [FLASK] Looking for files
flask_dev_local    | [FLASK]
flask_dev_local    | [FLASK] checking: Mitja_Jug.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Mitja_Jug.igc is a valid track
flask_dev_local    | [FLASK] checking: Igor_Eržen.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Igor_Eržen.igc is a valid track
flask_dev_local    | [FLASK] checking: Petros_Miskos.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Petros_Miskos.igc is a valid track
flask_dev_local    | [FLASK] checking: Slava_Jelić.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Slava_Jelić.igc is a valid track
flask_dev_local    | [FLASK] checking: Stjepan_Zjacic.IGC
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Stjepan_Zjacic.IGC is a valid track
flask_dev_local    | [FLASK] checking: Rafal_Cypcar.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Rafal_Cypcar.igc is a valid track
flask_dev_local    | [FLASK] checking: Jerko_Odžak.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Jerko_Odžak.igc is a valid track
flask_dev_local    | [FLASK] checking: Sandi_Krč.igc
flask_dev_local    | [FLASK] /tmp/tmp6autvfbu/Sandi_Krč.igc is a valid track
flask_dev_local    | [FLASK] with session id: 139980322766656
flask_dev_local    | [FLASK] Comp with registration: files will be checked against registered pilots not yet scored
flask_dev_local    | [FLASK] with session id: 139980322766656
flask_dev_local    | [FLASK] We have 6 pilots to find tracks for, and 8 tracks
flask_dev_local    | [FLASK] filename Mitja_Jug.igc, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Mitja', 'Jug']
flask_dev_local    | [FLASK] fai, Mitja
flask_dev_local    | [FLASK] name, Jug
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Mitja', 'Jug']
flask_dev_local    | [FLASK] Track Mitja_Jug.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Igor_Eržen.igc, <class 'str'>
flask_dev_local    | [FLASK] Track Igor_Eržen.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Petros_Miskos.igc, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Petros', 'Miskos']
flask_dev_local    | [FLASK] fai, Petros
flask_dev_local    | [FLASK] name, Miskos
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Petros', 'Miskos']
flask_dev_local    | [FLASK] Track Petros_Miskos.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Slava_Jelić.igc, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Slava', 'Jeli']
flask_dev_local    | [FLASK] fai, Slava
flask_dev_local    | [FLASK] name, Jeli
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Slava', 'Jeli']
flask_dev_local    | [FLASK] Track Slava_Jelić.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Stjepan_Zjacic.IGC, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Stjepan', 'Zjacic']
flask_dev_local    | [FLASK] fai, Stjepan
flask_dev_local    | [FLASK] name, Zjacic
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Stjepan', 'Zjacic']
flask_dev_local    | [FLASK] Track Stjepan_Zjacic.IGC is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Rafal_Cypcar.igc, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Rafal', 'Cypcar']
flask_dev_local    | [FLASK] fai, Rafal
flask_dev_local    | [FLASK] name, Cypcar
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Rafal', 'Cypcar']
flask_dev_local    | [FLASK] Track Rafal_Cypcar.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Jerko_Odžak.igc, <class 'str'>
flask_dev_local    | [FLASK] Track Jerko_Odžak.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] filename Sandi_Krč.igc, <class 'str'>
flask_dev_local    | [FLASK] ['fai', 'name']
flask_dev_local    | [FLASK] ['Sandi', 'Kr']
flask_dev_local    | [FLASK] fai, Sandi
flask_dev_local    | [FLASK] name, Kr
flask_dev_local    | [FLASK] ['name', 'name']
flask_dev_local    | [FLASK] ['Sandi', 'Kr']
flask_dev_local    | [FLASK] Track Sandi_Krč.igc is not a valid track file, pilot not found in competition or pilot already has a track
flask_dev_local    | [FLASK] *******************processed all tracks**********************

[philderbeast]

Looks like airScore is having trouble associating track logs with pilots. The message about invalidity is itself invalid because uploading these tracks individually works.

[kuaka]

So the output you have pasted is from the terminal not the processing log. Not that there is a lot of difference. The processing log will give you a bit more info. Looks like the first time through it found the tracks. Probably an issue with the quality. The processing log will hopefully give you more information

[philderbeast]

If you are running the 'prod' docker compose file it's the window that opens when processing igc files

I'm running docker-compose -f docker-compose-dev-local.yml up from the terminal. I do not see any windows opening.

Looks like the first time through it found the tracks. Probably an issue with the quality.

I have been able to upload the tracks individually. Wouldn't the quality assessment be the same either way?

[kuaka]

I'm running docker-compose -f docker-compose-dev-local.yml up from the terminal. I do not see any windows opening.

So by definition you are running Dev...

I have been able to upload the tracks individually. Wouldn't the quality assessment be the same either way?

Yes I think so

[kuaka]

I have tried to look at this but there is no fsdb file in the repo.

[philderbeast]

I moved to dot folders. The *.fsdb as XML is at .flight-system/clean-fsdb.xml. It got caught up in a sweeping change. I intend to move it and the other XML back to the root as the airScore open file dialog can't see those dot folders.

[kuaka]

dot folders are by default hidden in linux so not a great choice of location. running in prod on my system 8/8 tracks are uploaded.

Tracklog Processing. Track 8
Comp with registration: files will be checked against registered pilots not yet scored
We have 10 pilots to find tracks for, and 8 tracks
filename Sandi_Krč_22.igc,
processing 22 Sandi Krč:
***************START*******************
Pilot started SS at 12:34:11
Pilot landed after 3.49km
***************END****************
filename Petros_Miskos_18.igc,
processing 18 Petros Miskos:
***************START*******************
Pilot started SS at 12:34:55
Pilot landed after 2.98km
***************END****************
filename Stjepan_Zjacic_26.IGC,
processing 26 Stjepan Zjacic:
***************START*******************
Pilot started SS at 13:36:59
Pilot landed after 1.00km
***************END****************
filename Mitja_Jug_15.igc,
processing 15 Mitja Jug:
***************START*******************
Pilot started SS at 12:37:25
Pilot took TP01 at 13:09:34 at 687.0m
Pilot took TP02 at 13:39:09 at 1057.0m
Pilot landed after 11.55km
***************END****************
filename Slava_Jelić_24.igc,
processing 24 Slava Jelić:
***************START*******************
Pilot started SS at 12:45:07
Pilot landed after 0.99km
***************END****************
filename Rafal_Cypcar_19.igc,
processing 19 Rafal Cypcar:
***************START*******************
Pilot started SS at 12:36:10
Pilot took TP01 at 12:54:34 at 1129.0m
Pilot landed after 8.45km
***************END****************
filename Igor_Eržen_10.igc,
processing 10 Igor Eržen:
***************START*******************
Pilot started SS at 12:30:41
Pilot took TP01 at 12:44:52 at 0.0m
Pilot landed after 5.50km
***************END****************
filename Jerko_Odžak_11.igc,
processing 11 Jerko Odžak:
***************START*******************
Pilot started SS at 12:40:31
Pilot took TP01 at 13:07:01 at 0.0m
Pilot took TP02 at 13:50:11 at 0.0m
Pilot landed after 7.69km
***************END****************
*******************processed all tracks**********************`

suggest you try prod. If still not working check the filenames, it may be that there is an issue with your system and the unicode characters in the filenames.

[philderbeast]

I suspect you're right about the unicode characters.

You're not able to reproduce the problem on prod using files zipped on linux. I'm attaching files zipped using https://github.com/kuaka/fsdb_igc on osx, Task 1.zip and Task 2.zip. Do they work with your prod setup?

I'd like to stick to docker-compose-dev-local.yml if I can. Could you please try that too?

[kuaka]

You're not able to reproduce the problem on prod using files zipped on linux. I'm attaching files zipped using https://github.com/kuaka/fsdb_igc on osx, Task 1.zip and Task 2.zip. Do they work with your prod setup?

No. I tried task 1 and only found 4 tracks. However, I note that the zip files that you provided do not have the name_name_id.igc format. looks like you are running old fsdb_igc code? It was modified to add id in response to an issue you raised. In theory if airscore can't find a pilot by name it can use the id.

I'd like to stick to docker-compose-dev-local.yml if I can. Could you please try that too?

sure. same result as above.

Why are you hesitant to use prod? it literally takes seconds to switch between them (once prod docker has been built). I often switch to prod. It will give you a much better experience processing tracks (and full re scoring) and should not time out which can happen if you are processing tracks with the dev web server.

[kuaka]

Closing this as the original issue is due to users OS setup. This can be mitigated by importing files in _name_nameid.igc format. To automatically create this format the fsdb_igc tool can be used.