Import competition, forever processing.

[philderbeast]

I was given two *.fsdb from Igor Eržen, for 2018 and 2019 comps. I tried importing these but airscore hangs there processing and not completing the import.

The server log is showing a 500 response:

flask_dev    | "POST /users/_import_comp_fsdb/ HTTP/1.1" 500 -

Traceback (most recent call last):
...
  File "/app/airscore/user/views.py", line 166, in _import_comp_fsdb
    f = FSDB.read(fsdb_file)
  File "/app/airscore/core/fsdb.py", line 83, in read
    pilot = Participant.from_fsdb(pil, from_CIVL=from_CIVL)
  File "/app/airscore/core/pilot/participant.py", line 134, in from_fsdb
    live = next(el for el in childs if el.get('name') == 'Live')
StopIteration

[philderbeast]

This happens with the original *.fsdb but not with the *.clean-fsdb.xml renamed to *.fsdb for Greda 2019. Same goes for Greda 2018.

[kuaka]

I have the same issue with the file provided in #157

[biuti]

I should have fixed original problem with https://github.com/FAI-CIVL/FAI-Airscore/commit/61286993b7a649991dcfc908d7b94c016232df06. Exported file has different problems, mainly a lot of None values. I had also an error trying to import: File "/app/airscore/core/formula.py", line 438, in get_fsdb_info formula.score_back_time = 0 + int(form.get('score_back_time')) * 60 # Scoreback Time, seconds ValueError: invalid literal for int() with base 10: '15.0' I'll look into it.

[biuti]

I should have fixed importing and exporting; Now it should export a simplified file without None values. It is importing correctly formula as well, as far as parameters are recognised. Commit https://github.com/FAI-CIVL/FAI-Airscore/commit/acbfbd60f2d4a559a405f97bb420810e9ffe50c4 Needs previous commit as well that enhances save_or_update() Model method to cover case when we have a new row to be added that already has a primary key value.

[biuti]

Version created after commit. DALBIG17_2_20201118_024747.fsdb.zip

[philderbeast]

I tested against a279e3e29077590ec5d4f9501b601b6e62912644. On importing the Dalmatian 2019 comp, I get:

Traceback (most recent call last):
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sid/.local/lib/python3.8/site-packages/flask_debugtoolbar/__init__.py", line 125, in dispatch_request
    return view_func(**req.view_args)
  File "/home/sid/.local/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    return func(*args, **kwargs)
  File "/app/airscore/user/views.py", line 186, in _import_comp_fsdb
    compid = f.add_all()
  File "/app/airscore/core/fsdb.py", line 651, in add_all
    self.create_results_files()
  File "/app/airscore/core/fsdb.py", line 630, in create_results_files
    ref_id, filename, _ = create_json_file(comp_id=self.comp.comp_id, task_id=task.id,
  File "/app/airscore/core/result.py", line 548, in create_json_file
    with open(RESULTDIR + filename, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/app/airscore/data/results/DALXC19_6_T1_20201125_145701.json'

[kuaka]

@biuti I believe this is because if we start with a fresh install of airscore and import a comp then the data/result/ dir does not exist and this error will happen. With a normal comp we are checking that it exists and creating if not (I think). Need to have a check either in result file saving or fsdb function or probably best at app startup. Check all defines dirs and create if not existing. that way will cover all scenarios.

[biuti]

added a functions that creates folders at app start if not existing (https://github.com/FAI-CIVL/FAI-Airscore/commit/c70d00fb3fee35e1fa5347805949f681fe613584)

[philderbeast]

@biuti, I did a fresh clone at 6173c7952622d402375415fe43689cb8a1a39abb and checked git's log, finding c70d00fb3fee35e1fa5347805949f681fe613584 is included on the development branch.

Using docker-compose -f docker-compose-dev-local.yml up and importing Greda 2019, on bulk importing tracks for task 1, I find that an expected directory is missing.

Traceback (most recent call last):
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/sid/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/sid/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sid/.local/lib/python3.8/site-packages/flask_debugtoolbar/__init__.py", line 125, in dispatch_request
    return view_func(**req.view_args)
  File "/home/sid/.local/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    return func(*args, **kwargs)
  File "/app/airscore/user/views.py", line 983, in _upload_track_zip
    resp = frontendUtils.process_zip_file(zip_file=zip_file,
  File "/app/airscore/core/frontendUtils.py", line 909, in process_zip_file
    data = process_archive(task, zip_file, check_g_record=grecord, track_source=track_source)
  File "/app/airscore/core/frontendUtils.py", line 887, in process_archive
    assign_and_import_tracks(tracks, task, track_source, check_g_record=check_g_record)
  File "/app/airscore/core/trackUtils.py", line 172, in assign_and_import_tracks
    verify_and_import_track(pilot, mytrack, task, print=new_print)
  File "/app/airscore/core/trackUtils.py", line 187, in verify_and_import_track
    result.save_tracklog_map_file(task, track.flight)
  File "/app/airscore/core/pilot/flightresult.py", line 493, in save_tracklog_map_file
    res_path.mkdir(mode=0o755)
  File "/usr/local/lib/python3.8/pathlib.py", line 1279, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/app/airscore/data/map-objects/tracks/1'

The folders on disk at airscore/data on my disk mapped to /app/airscore under docker after this failure:

> tree -d
.
├── airspace
│   ├── check_file
│   └── openair
├── livetracking
├── log
├── map-objects
│   └── airspace
├── results
├── temp
├── tracks
│   └── 2019
│       └── dalxc19
│           ├── t1_20190711
│           └── t2_20190712
└── waypoints

[philderbeast]

Fixed by https://github.com/FAI-CIVL/FAI-Airscore/pull/222#issuecomment-750939753.