Ingestion API is stuck for ModFlow results

dgarijo commented 4 years ago

Describe the bug I created a single run for ModFlow and it's been 20 min ingesting the data with no result: https://mint.isi.edu/texas/modeling/scenario/jPK2p6LwSbLXbWRjQsP2/4zhf8ov8wPw7r4SlDgw4/FtA5fsVmvMmwzN4Zi7SF/results The results are big (1.5 GB), but the result table should be fairly simple.

mosoriob commented 4 years ago

The ingestion is trying to insert into the table modflow_2005_cfg_lst

The column names of the table are

     Table "public.modflow_2005_cfg_lst"
    Column    |       Type       | Modifiers
--------------+------------------+-----------
 threadid     | text             |
 mint_runid   | text             |
 unnamed:_0   | double precision |
 unnamed:_1   | text             |
 unnamed:_2   | text             |
 unnamed:_3   | text             |
 unnamed:_4   | text             |
 unnamed:_5   | text             |
 unnamed:_6   | text             |
 unnamed:_7   | text             |
 unnamed:_8   | text             |
 unnamed:_9   | text             |
 unnamed:_10  | text             |
 unnamed:_11  | text             |
 unnamed:_12  | text             |
 unnamed:_13  | text             |
 unnamed:_14  | text             |
 unnamed:_15  | text             |
 unnamed:_16  | text             |
 unnamed:_17  | text             |
 unnamed:_18  | text             |
 unnamed:_19  | text             |
 unnamed:_20  | text             |
 unnamed:_21  | text             |
 unnamed:_22  | text             |
 unnamed:_23  | text             |
 unnamed:_24  | text             |
 unnamed:_25  | text             |
 unnamed:_26  | text             |
 unnamed:_27  | text             |
 unnamed:_28  | text             |
 unnamed:_29  | text             |
 unnamed:_30  | text             |
 unnamed:_31  | text             |
 unnamed:_32  | text             |
 unnamed:_33  | text             |
 modflow-2005 | text             |
 unnamed:_35  | text             |
 unnamed:_36  | text             |
 unnamed:_37  | text             |
 unnamed:_38  | text             |
 unnamed:_39  | text             |

These names are weird

 16713 | devingestion | 21562 |    16384 | ingestion |                        | 128.9.44.22    |                 |       57396 | 2020-02-27 01:04:41.397952+00 | 2020-02-27 01:0
9:00.475891+00 | 2020-02-27 01:33:15.680115+00 | 2020-02-27 01:33:15.680206+00 | f       | idle in transaction |      185218 |              | INSERT INTO modflow_2005_cfg_lst (
threadid, mint_runid, "unnamed:_0", "unnamed:_1", "unnamed:_2", "unnamed:_3", "unnamed:_4", "unnamed:_5", "unnamed:_6", "unnamed:_7", "unnamed:_8", "unnamed:_9", "unnamed:_10",
 "unnamed:_11", "unnamed:_12", "unnamed:_13", "unnamed:_14", "unnamed:_15", "unnamed:_16", "unnamed:_17", "unnamed:_18", "unnamed:_19", "unnamed:_20", "unnamed:_21", "unnamed:_
22", "unnamed:_23", "unnamed:_24", "unnamed:_25", "unnamed:_26", "unnamed:_27", "unnamed:_28", "unnamed:_29", "unnamed:_30", "unnamed:_31", "unnamed:_32", "unnamed:_33", "modfl
ow-2005", "unnamed:_35", "unnamed:_36", "unnamed:_37", "unnamed:_38", "unnamed:_39") VALUES ('FtA5fsVmvMmwzN4Zi7SF', '20424b08aa37f87a5c54e3a59315443a', NULL, NULL, NULL, NULL,
 '876', NULL, NULL, NULL, NULL, NULL, '1', NULL, NULL, NULL, NULL, '18', NULL, NULL, NULL, NULL, '58', NULL, NULL, NULL, NULL, NULL, '-14.90', NULL, NULL, NULL, NULL, NULL, NUL
L, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
 16713 | devingestion | 22157 |    16384 | ingestion |                        | 128.9.44.22    |                 |       57728 | 2020-02-27 01:13:39.079215+00 | 2020-02-27 01:1
8:13.543644+00 | 2020-02-27 01:33:15.691004+00 | 2020-02-27 01:33:15.691087+00 | f       | idle in transaction |      185222 |              | INSERT INTO modflow_2005_cfg_lst (
threadid, mint_runid, "unnamed:_0", "unnamed:_1", "unnamed:_2", "unnamed:_3", "unnamed:_4", "unnamed:_5", "unnamed:_6", "unnamed:_7", "unnamed:_8", "unnamed:_9", "unnamed:_10",
 "unnamed:_11", "unnamed:_12", "unnamed:_13", "unnamed:_14", "unnamed:_15", "unnamed:_16", "unnamed:_17", "unnamed:_18", "unnamed:_19", "unnamed:_20", "unnamed:_21", "unnamed:_
22", "unnamed:_23", "unnamed:_24", "unnamed:_25", "unnamed:_26", "unnamed:_27", "unnamed:_28", "unnamed:_29", "unnamed:_30", "unnamed:_31", "unnamed:_32", "unnamed:_33", "modfl
ow-2005", "unnamed:_35", "unnamed:_36", "unnamed:_37", "unnamed:_38", "unnamed:_39") VALUES ('krtbF2nieT34QJWTtxCO', 'ed58ec26b3bb31a4eec7c50febacd0f9', NULL, NULL, NULL, NULL,
 NULL, NULL, NULL, '5.4651E-02', NULL, '0.1728', NULL, NULL, NULL, NULL, NULL, NULL, '1.189', NULL, NULL, NULL, NULL, NULL, NULL, '5.606', NULL, NULL, NULL, NULL, NULL, NULL, '
11.87', NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL)

mosoriob commented 4 years ago

I executed the query

INSERT INTO modflow_2005_cfg_lst (threadid, mint_runid, "unnamed:_0", "unnamed:_1", "unnamed:_2", "unnamed:_3", "unnamed:_4", "unnamed:_5", "unnamed:_6", "unnamed:_7", "unnamed:_8", "unnamed:_9", "unnamed:_10", "unnamed:_11", "unnamed:_12", "unnamed:_13", "unnamed:_14", "unnamed:_15", "unnamed:_16", "unnamed:_17", "unnamed:_18", "unnamed:_19", "unnamed:_20", "unnamed:_21", "unnamed:_22", "unnamed:_23", "unnamed:_24", "unnamed:_25", "unnamed:_26", "unnamed:_27", "unnamed:_28", "unnamed:_29", "unnamed:_30", "unnamed:_31", "unnamed:_32", "unnamed:_33", "modflow-2005", "unnamed:_35", "unnamed:_36", "unnamed:_37", "unnamed:_38", "unnamed:_39") VALUES ('FtA5fsVmvMmwzN4Zi7SF', '20424b08aa37f87a5c54e3a59315443a', NULL, NULL, NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL, NULL, NULL, '-1000.', NULL, NULL, NULL, NULL, NULL, NULL, '503.7', NULL)

And it works

mosoriob commented 4 years ago

@varunratnakar I'm not familiar with ingestion. Can you work on it?

mosoriob commented 4 years ago

Do we have ingest the download links?

mosoriob commented 4 years ago

I'm going to close this issue due to inactivity

mintproject / mint-ui-lit

Ingestion API is stuck for ModFlow results #303