Seems that at least something wrong with send_and_recive_and_write function.
The connection with phobos is successfully closed, but that with mars is not. Thus, we can conclude that send_request is properly finished. At closer look, we can find that data is successfully received from both mars and phobos servers:
[DEBUG] 2023-03-03 01:54:07,056 hifuku.datagen.http_datagen.request: got renpose from (mars, 8080)
[DEBUG] 2023-03-03 01:54:07,058 hifuku.datagen.http_datagen.request: raw response object sizes: 2852917557 byte
[DEBUG] 2023-03-03 01:54:43,833 hifuku.datagen.http_datagen.request: got renpose from (phobos, 8080)
[DEBUG] 2023-03-03 01:54:43,835 hifuku.datagen.http_datagen.request: raw response object sizes: 2852980400 byte
3/2
so we can conclude that something wrong with depickling the raw contents
There could be two causes:
due to data size (like memory error, storage limit)
due to low-probability failure case that become visible due to the large size
3/3
we found that the program does not enter exception handling procedure, which implies the program immediately terminated in the emergent situation such as memory error. This hypothesis is strengthen by the fact that the de-pickling always failed after processing second data (e.g. data for phobos is completed. and then data for mars proceeds)
[DEBUG] 2023-03-03 01:25:06,371 hifuku.datagen.batch_solver: run self.solve_batch
[INFO] 2023-03-03 01:25:06,371 hifuku.datagen.http_datagen.client: n_gen_table: {('phobos', 8080): 5000, ('mars', 8080): 5000}
[DEBUG] 2023-03-03 01:25:06,623 hifuku.datagen.batch_solver: send_and_recive_and_write called on pid: 3364395
[DEBUG] 2023-03-03 01:25:06,623 hifuku.datagen.http_datagen.request: try to connect ('phobos', 8080)
[DEBUG] 2023-03-03 01:25:06,624 hifuku.datagen.http_datagen.request: connected to ('phobos', 8080)!
[DEBUG] 2023-03-03 01:25:06,893 hifuku.datagen.batch_solver: send_and_recive_and_write called on pid: 3364396
[DEBUG] 2023-03-03 01:25:06,893 hifuku.datagen.http_datagen.request: try to connect ('mars', 8080)
[DEBUG] 2023-03-03 01:25:06,893 hifuku.datagen.http_datagen.request: connected to ('mars', 8080)!
[DEBUG] 2023-03-03 01:25:21,047 hifuku.datagen.http_datagen.request: elapsed time to serialize: 14.423235177993774
[DEBUG] 2023-03-03 01:25:21,049 hifuku.datagen.http_datagen.request: serialized object sizes: 3798459112 byte
[DEBUG] 2023-03-03 01:25:21,107 hifuku.datagen.http_datagen.request: elapsed time to serialize: 14.213094234466553
[DEBUG] 2023-03-03 01:25:21,109 hifuku.datagen.http_datagen.request: serialized object sizes: 3798459226 byte
[DEBUG] 2023-03-03 01:28:13,948 hifuku.datagen.http_datagen.request: send request to (mars, 8080): [...hogehoge...]
[DEBUG] 2023-03-03 01:28:13,949 hifuku.datagen.http_datagen.request: elapsed time to send request: 172.838552236557
[DEBUG] 2023-03-03 01:28:17,617 hifuku.datagen.http_datagen.request: send request to (phobos, 8080): [...hogehoge...]
[DEBUG] 2023-03-03 01:28:17,618 hifuku.datagen.http_datagen.request: elapsed time to send request: 176.56715178489685
[DEBUG] 2023-03-03 01:54:07,056 hifuku.datagen.http_datagen.request: got renpose from (mars, 8080)
[DEBUG] 2023-03-03 01:54:07,058 hifuku.datagen.http_datagen.request: raw response object sizes: 2852917557 byte
[DEBUG] 2023-03-03 01:54:43,833 hifuku.datagen.http_datagen.request: got renpose from (phobos, 8080)
[DEBUG] 2023-03-03 01:54:43,835 hifuku.datagen.http_datagen.request: raw response object sizes: 2852980400 byte
[DEBUG] 2023-03-03 01:55:27,465 hifuku.datagen.http_datagen.request: elapsed time to deserialize: 43.62995147705078
[DEBUG] 2023-03-03 01:55:27,470 hifuku.datagen.http_datagen.request: response contents: [...hogehoge...]
[DEBUG] 2023-03-03 01:55:27,699 hifuku.datagen.http_datagen.request: closed connectionto ('phobos', 8080)!
[DEBUG] 2023-03-03 01:56:49,844 hifuku.datagen.batch_solver: saved to /tmp/tmpt6bl4_iy/f00d3b3f-bebf-4f25-8287-758769a76287
[DEBUG] 2023-03-03 01:56:49,844 hifuku.datagen.batch_solver: send_and_recive_and_write finished on pid: 3364395
Seems that at least something wrong with
send_and_recive_and_writefunction. The connection withphobosis successfully closed, but that with mars is not. Thus, we can conclude thatsend_requestis properly finished. At closer look, we can find that data is successfully received from bothmarsandphobosservers:3/2 so we can conclude that something wrong with depickling the raw contents There could be two causes:
due to low-probability failure case that become visible due to the large size3/3 we found that the program does not enter exception handling procedure, which implies the program immediately terminated in the emergent situation such as memory error. This hypothesis is strengthen by the fact that the de-pickling always failed after processing second data (e.g. data for phobos is completed. and then data for mars proceeds)