search

qiboteam / qibolab

Quantum hardware module and drivers for Qibo.
https://qibo.science
Apache License 2.0
40 stars 10 forks source link

Qblox Timeout while waiting for instrument response (100)', cmd='SLOT18:SEQuencer0:STATE?')" #845

Open aorgazf opened 3 months ago

aorgazf commented 3 months ago

Dear @hay-k As discussed, I've spotted an error that seems to have been triggered by a change in the driver of qblox. The error message reads: 

RuntimeError: "Execution error;(reason='module command SEQ#:STATE? failed on module in slot 18: Timeout while waiting for instrument response (100)', cmd='SLOT18:SEQuencer0:STATE?')"

If you compare the debug folders, you can see that in the new version of the driver, sequence programs for idle sequencers are not generated, however it appears that despite of being idle, those sequencers have been configured to sync with the rest, and without a sequencer program to execute, the whole instrument is stuck waiting for those sequencers sync instructions. Eventually, he instrument times out. image

To reproduce this error, one needs to work simultaneously with a number of qubits that does not use all the outputs in all the baseband control modules. (I'm not sure if a similar issue exists with drive/readout pulses).

20240320 qblox driver issues.zip

I've recreated branches in qibocal and qibolab from 15/12/2023: alvaro/latest_20231215 And I've included old platform & action runcards in the zip file to facilitate testing with the old branches.

hay-k commented 3 months ago

Investigation reveals that this issue is not related to the instrument waiting forever for unused sequencers to sync. The issue is in the communication of the driver with the instrument. When 5 sequencers are used, it looks like the instrument is too busy and is not able to respond to the state request on time, hence the request times out. Unfortunately, the actual function that times out does not have a configurable timeout (and it is part of the qblox-instruments package, so I cannot quickly change it). I will try to put up a quick workaround in our driver for now and also discuss with them to see what is the best way to go forward with this.

hay-k commented 3 months ago

Short status update: I have now narrowed this issue down to a small self-contained reproducible case. There is also confirmation that the issue can be successfully reproduced on other instrument clusters, so it should not be specific to our lab setup. Continuing to collaborate with qblox developers to further troubleshoot the issue.

hay-k commented 3 months ago

Until the issue is fixed, the most reliable thing to do to avoid it is not use more than 4 sequencers of the readout module simultaneously. From the perspective of calibration experiments in our lab this translates into not doing experiments with more than 4 qubits at once.

P.S. There are other workarounds, but this one is the most reliable one.