Not creation of the ascci or parquet point data table in version 0.11.2

[FatimaPillosu]

When we ask the software to start creating the point data table, two error messages appear depending on whether we are creating an ascii or a parquet table:

ASCII PDT

In this case, the computations seem to not even start! Note that @AugustinVintzileos got my same errors.

omicron{/home/mo/mofp/Downloads}:10 --> ./ecPoint-Calibrate-0.11.2.AppImage Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank" Pulling from onyb/ecpoint-calibrate-logger Digest: sha256:2238a96fe114e3a8dc638175ed9c92d336da6b3b73394d6786554f9201b27ea9 Status: Image is up to date for onyb/ecpoint-calibrate-logger:0.11.2 Pulling from onyb/ecpoint-calibrate-core Digest: sha256:0ae4d7239b752d85969533c3b1e116af09bd98564af7374bfc2c8e9f844c1cf1 Status: Image is up to date for onyb/ecpoint-calibrate-core:0.11.2 Running Docker container: image=onyb/ecpoint-calibrate-logger:0.11.2 containerID=9f2060bdb854 Running Docker container: image=onyb/ecpoint-calibrate-core:0.11.2 containerID=ef0d0d2af727

• Serving Flask app "api" (lazy loading)
• Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
• Debug mode: off
• Running on http://0.0.0.0:8888/ (Press CTRL+C to quit) Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 21:31:08] "POST /get-predictor-units HTTP/1.1" 200 - Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 21:31:41] "POST /predictors HTTP/1.1" 200 - Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 21:32:22] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:33:20] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:43:04] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:53:43] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:09] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:10] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:10] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:29] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:34] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:35] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:35] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:54:55] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:55:24] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:55:44] "POST /get-predictor-units HTTP/1.1" 200 - 172.17.0.1 - - [04/Nov/2020 21:55:54] "POST /get-predictor-units HTTP/1.1" 200 - Exception on /computation-logs [POST] Traceback (most recent call last): File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/app/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function return cors_after_request(app.make_response(f(*args, kwargs))) File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/app/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](req.view_args) File "/app/core/api.py", line 34, in stream_computation_logs run(config) File "/app/core/processor/init.py", line 71, in run serializer.add_header(header.strip()) File "/app/core/loaders/ascii.py", line 18, in add_header with open(self.path, "w") as f: FileNotFoundError: [Errno 2] No such file or directory: '/home/mo/mofp/out.ascii' 172.17.0.1 - - [04/Nov/2020 22:01:25] "POST /computation-logs HTTP/1.1" 500 -

---

PARQUET TABLE

In this case, the computations seem to start correctly but when we need to save the results in the file, something goes wrong.

^Comicron{/home/mo/mofp/Downloads}:11 --> ./ecPoint-Calibrate-0.11.2.AppImage Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank" Stopping container: ef0d0d2af727 Stopping container: 9f2060bdb854 Pulling from onyb/ecpoint-calibrate-core Digest: sha256:0ae4d7239b752d85969533c3b1e116af09bd98564af7374bfc2c8e9f844c1cf1 Status: Image is up to date for onyb/ecpoint-calibrate-core:0.11.2 Pulling from onyb/ecpoint-calibrate-logger Digest: sha256:2238a96fe114e3a8dc638175ed9c92d336da6b3b73394d6786554f9201b27ea9 Status: Image is up to date for onyb/ecpoint-calibrate-logger:0.11.2 Running Docker container: image=onyb/ecpoint-calibrate-core:0.11.2 containerID=a0d92c71ad75 Running Docker container: image=onyb/ecpoint-calibrate-logger:0.11.2 containerID=7ba63efdc072

• Serving Flask app "api" (lazy loading)
• Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
• Debug mode: off
• Running on http://0.0.0.0:8888/ (Press CTRL+C to quit) Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 22:09:13] "POST /get-predictor-units HTTP/1.1" 200 - Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 22:09:20] "POST /get-predictor-units HTTP/1.1" 200 - Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 22:09:36] "POST /predictors HTTP/1.1" 200 - Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. 172.17.0.1 - - [04/Nov/2020 22:09:56] "POST /get-predictor-units HTTP/1.1" 200 -

---

ecPoint-Calibrate - POINT DATA TABLE

---

GENERAL PARAMETERS Model Data Format = GRIB Start Calibration Period = 2019-01-01 End Calibration Period = 2019-01-01 First Model Run (in the day) = 0 UTC Last Model Run (in the day) = 12 UTC Interval between Model Runs = 12 h Interval between Forecast Validity Time = 1 h Forecast Data Sampling Interval = 1 h Spin-Up Window = 3 h

PREDICTAND Variable = tp (in m) Type = Accumulated
Accumulation = 12 h Minimum Value = 0.001 m Scaling Factor (Multiply) = 1000.0 Scaling Factor (Add) = 0.0 Forecast Database = /vol/ecpoint/mofp/calibrations/InputDB/Rainfall/ECMWF_ENS_47r1/FC_new/tp Forecase Error = Forecast Error Ratio (FER) [-]

PREDICTORS Forecast Database = /vol/ecpoint/mofp/calibrations/InputDB/Rainfall/ECMWF_ENS_47r1/FC_new List of Predictors:

• TP, TP [mm]

OBSERVATIONS Parameter = tp (in mm) Observations Database = /vol/ecpoint/mofp/calibrations/InputDB/Rainfall/ECMWF_ENS_47r1/OBS

OUTPUT FILE Path = /home/mo/mofp/Downloads/out.parquet

---

ecPoint-Calibrate - POINT DATA TABLE

---

NOTE: 'DateOBS' and 'TimeOBS' correspond to the date and time that the observation was taken. (as defined by 'BaseDate', 'BaseTime', and 'StepF').

  'BaseDate' and 'DateOBS' are given in the YYYY-MM-DD format.
  'BaseTime' and 'TimeOBS' are given in UTC time.
  'StepF' is given in the HH format.
  'LatOBS' is given in degrees North (i.e. from -90 to +90 N).
  'LonOBS' is given in degrees East (i.e. from -180 to +180 E).

START COMPUTATIONS

The start step of the acc period (StepS) is considered between 'LimSU' (to avoid the spin-up window), and 'IntBT+(LimSU-1)' (to consider the shortest range forecast available)."

Therefore, StepS will range between t+3 and t+14

Case 1 FORECAST PARAMETERS: 2019-01-01, 00 UTC, (t+3, t+15)

OBSERVATIONS PARAMETERS: Validity date/time (end of 12 h period) = 2019-01-01 15:00:00 Read observation file: tp_12_20190101_15.geo

PREDICTORS COMPUTATIONS: Reading forecast file: tp_20190101_00_03.grib Reading forecast file: tp_20190101_00_15.grib Computing TP using 2 input(s). Selecting the nearest grid point to observations. Selecting values that correspond to TP >= 1.0 mm/12h.

Computing the FER.

POINT DATA TABLE: Saving the point data table to output file: /home/mo/mofp/Downloads/out.parquet Point data table format: PARQUET Exception on /computation-logs [POST] Traceback (most recent call last): File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/app/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function return cors_after_request(app.make_response(f(*args, kwargs))) File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/app/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](req.view_args) File "/app/core/api.py", line 34, in stream_computation_logs run(config) File "/app/core/processor/init.py", line 487, in run serializer.add_columns_chunk(columns) File "/app/core/loaders/parquet.py", line 79, in add_columns_chunk return self.append(df) File "/app/core/loaders/parquet.py", line 62, in append self._pq_writer = pq.ParquetWriter(f"{self.path}", self._schema) File "/app/.venv/lib/python3.8/site-packages/pyarrow/parquet.py", line 549, in init sink = self.file_handle = filesystem.open_output_stream(path) File "pyarrow/_fs.pyx", line 660, in pyarrow._fs.FileSystem.open_output_stream File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status FileNotFoundError: [Errno 2] Failed to open local file '/home/mo/mofp/Downloads/out.parquet'. Detail: [errno 2] No such file or directory 172.17.0.1 - - [04/Nov/2020 22:10:11] "POST /computation-logs HTTP/1.1" 500 -

[FatimaPillosu]

Update on this issue. The software works (reads and save files) only if we work in /vol. THis is confusing becuase as we can see all the file system, one might decide to save the files somewhere else, but he can't. How can we solve the problem?

[onyb]

Is this issue still valid?