Closed enricorotundo closed 6 months ago
More info from Mihirsinh Parmar (link to slack thread here):
Hey, the docker image mihirsinhparmar/bac was deleted, as it was giving error and even when I updated the docker image, bacalhau was still running old docker container, so I moved to a different docker container name mihirsinhparmar/bacalhau-ie This one works on bacalhau. Though I have to try it by providing inputs.
I believe, bacalhau waiting at Creating job for submission because IPFS hash was not propagated to filecoin. I pinned the file using Web3Storage. But I recently checked the file is still in queue. I pinned a file few days ago. the file: bafybeidnwj4wlfsshgsfbuinelg5hsovlg6ibozyzyii7uw6dwgv6lytcm
I just uploaded a new file today with web3 storage. It is still queuing, can this IPFS hash be used for inputs or does the filecoin deal should be made before the IPFS hash can be used with bacalhau.
Issue
Originally raised in this Slack thread, the following command
bacalhau docker run -v bafybeidnwj4wlfsshgsfbuinelg5hsovlg6ibozyzyii7uw6dwgv6lytcm:/inputs mihirsinhparmar/bac
reportedly got stuck at "Creating job for submission ..." (on v0.3.16), the report indicatesc90bad95-8202-4638-96a0-4e09a545f180
as the related job-id.There're suspiciously too few event logs related to that job-id. These were recorded between
2023-01-02T16:40:00
and2023-01-02T16:45:00
:Furthermore, it seems there're no logs pointing to that job-id around the time of the incident. This is werid.
Investigation
Two things to note here:
mihirsinhparmar/bac
is not an existing image (check on Docker Hub). We can probably rule out this option because although https://github.com/filecoin-project/bacalhau/issues/1587 and https://github.com/filecoin-project/bacalhau/issues/1589, the submit command should still return quickly with a genericError while executing the job
.Now, the log
Error error completing job due to shard timed out while in state EnqueuingBids, see --timeout
appears in both my repro and in the reported issue so I'd say the root cause is probably a silent timeout whilst looking up a CID in IPFS.