InferenceServerException: [StatusCode.UNKNOWN] Stream removed

[colin986]

Good morning,

I have been using Oktoberfest successfully for a number of months to generate spectral libraries with Prosit - great tool!

Yesterday, I was attempting to generate a new library and received the following error when using both docker and Oktoberfest in a conda environment (both different versions).

Error output:

Inferring predictions for 7000 spectra with batch site 1000: 86%|████████▌ | 6/7 [00:05<00:01, 1.02s/it]E0105 09:58:07.043478763 1 hpack_parser.cc:999] Error parsing 'content-type' metadata: invalid value Inferring predictions for 7000 spectra with batch site 1000: 86%|████████▌ | 6/7 [01:05<00:10, 10.94s/it] Traceback (most recent call last): File "", line 1, in File "/root/oktoberfest/runner.py", line 391, in run_job generate_spectral_lib(config_path) File "/root/oktoberfest/runner.py", line 195, in generate_spectral_lib grpc_output_sec = pr.grpc_predict(spectra_div, **server_kwargs) File "/root/oktoberfest/predict/predict.py", line 63, in grpc_predict intensity_predictions = infer_predictions( File "/root/oktoberfest/predict/predict.py", line 153, in infer_predictions prediction = triton_client.infer(model, inputs=infer_inputs, outputs=infer_outputs) File "/usr/local/lib/python3.8/site-packages/tritonclient/grpc/_client.py", line 1321, in infer raise_error_grpc(rpc_error) File "/usr/local/lib/python3.8/site-packages/tritonclient/grpc/_utils.py", line 62, in raise_error_grpc raise get_error_grpc(rpc_error) from None tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] Stream removed

Operating system: Linux Ubuntu Octoberfest docker version: 0.5.1 Oktoberfest conda version: 0.5.0

If you would like to replicate the error the prosit input and config.json I used are attached.

prosit_input.csv config.json

[colin986]

I created a new docker container with oktoberfest 0.5.3, which seems to have fixed the error I was getting.

However I now get the following warning:

Getting predictions: 100%|██████████| 1/1 [00:00<00:00, 1.66it/s] /root/oktoberfest/predict/predict.py:43: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy data.rename( /root/oktoberfest/predict/predict.py:31: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

[picciama]

Hi @colin986 , Yes, there is un issue with spectral library generation when there is a failure in getting predictions. This can be due to a short disconnect to the server. I am currently fixing the last pieces of a new version that solves many issues, which simply skips a batch that failed and continues with the rest, so that you can simply rerun and it only appends the failed batches to the existing library so that you don't have to restart it. The copy warning is another problem that I fixed and which will be part of the next release. I am hopeful to releasing by the end of the week.

[colin986]

Thanks @picciama. I used a hacky solution from my side to split my Fasta into chunks and loop through - I then check the output and if needed run the chunk again.

I'll keep an eye out for the new update.

[picciama]

@colin986 The current development branch contains a completely rewritten spectral library workflow in case you want to test it prior to the next stable release. It is way faster, offers more options and is more stable: You can now simply restart the run if the prediction of some batches failed, as long as you don't make any change to the config file and leave it run to ensure the writer finishes without corrupting the file. The log output was improved accordingly.

Please check the new documentation, since this is a breaking change in the config file: https://oktoberfest.readthedocs.io/en/latest/jobs.html#b-spectral-library-generation.

In case you are interested in some more technical considerations when running this, please have a look at this very detailed comment: https://github.com/wilhelm-lab/oktoberfest/issues/126#issuecomment-1890152806

[colin986]

Hi @picciama

I have tested the implementation and can confirm that everything is OK. Thanks a million!

Colin

[picciama]

The new oktoberfest version 0.6.0 is published.