Closed Mariano215 closed 3 years ago
Device event pipeline takes awhile to run and populate the index.
It's been up for about 6 hours and 30 minutes and no data in device events so far. I'm running 8 processors with 54 G Ram and 2 T drive dedicated to the virtual machine.
Is there a way to check what's happening?
Latest status: pipeline_1 | 2020-12-22 17:27:42,345 mapreduce.py:176 Starting MapReduce: [Collection(54 items)] -> ./data/maude/init-2020-12-20-json.db@2, M: CSV2JSONMapper, R: CSV2JSONJoinReducer pipeline_1 | 2020-12-22 21:27:42,574 pipeline.py:468 Does not conform to device structure. Skipping: 7763969, ##### pipeline_1 | 2020-12-22 21:27:42,575 pipeline.py:468 Does not conform to device structure. Skipping: 4201358, ##### pipeline_1 | 2020-12-22 21:27:48,052 pipeline.py:468 Does not conform to device structure. Skipping: 7773558, ##### pipeline_1 | 2020-12-22 21:27:48,052 pipeline.py:468 Does not conform to device structure. Skipping: 1902177, ##### pipeline_1 | 2020-12-22 21:27:55,544 pipeline.py:468 Does not conform to device structure. Skipping: 7780933, ##### pipeline_1 | 2020-12-22 21:27:55,545 pipeline.py:468 Does not conform to device structure. Skipping: 3501120, #####
curl http://localhost:8000/status [ { "endpoint": "deviceclearance", "status": "GREEN", "last_updated": "2020-12-14", "documents": 157716, "requests": 20, "latency": 6.619047619047619 }, { "endpoint": "foodevent", "status": "GREEN", "last_updated": "2020-07-28", "documents": 91776, "requests": 20, "latency": 5.761904761904762 }, { "endpoint": "othernsde", "status": "GREEN", "last_updated": "2020-12-19", "documents": 469657, "requests": 20, "latency": 28.238095238095237 }, { "endpoint": "othersubstance", "status": "GREEN", "last_updated": "2020-12-11", "documents": 119479, "requests": 20, "latency": 27.476190476190474 } ]
Now it errored out...
pipeline_1 | Traceback (most recent call last):
pipeline_1 | File "openfda/maude/pipeline.py", line 774, in
This particular pipeline does a lot of data crunching for quite a while (index remains empty) and then writes results to the Elasticsearch index as the very last step. If it has not yet finished on your end, please feel free to share the container log file.
OK, so it did finish but with an error. I'm happy to look at your log files:docker-compose logs --tail="all" pipeline
Much appreciated: docker-logs.zip
Some other questions if you don't mind:
Thanks and I look forward to your analysis - happy to help in any way I can.
leveldb.LevelDBError: IO error: ./data/maude/init-2020-12-20-json.db-mapreduce-output-2020-12-22-05-27/shard-00001-of-00002.db: Too many open files
Ah. Too many open files. Let us adjust the docker setup and do a PR with the fix. Please stay tuned.
Attaching docker compose logs. docker_compose.zip