Unexpected return value from NetMHCstabpan server

[asmlgkj]

Thanks a lot.

after change the method from NetMHCpan to MHCnuggetsI
command
docker run --rm --user id -u:id -g -v $4:/data docker.io/griffithlab/pvactools pvacseq run /data/$3 $1 ${content_hla} MHCnuggetsI /data/pvacseq -e1 8,9,10,11 --iedb-install-directory /opt/iedb -t 4 --tdna-vaf 0.01 --normal-sample-name $1_N -k --downstream-sequence-length 1000 -c 1 --exclude-NAs --netmhc-stab --net-chop-method cterm
the cannot fork disappers, but I still do not know how to solve the problem.
but new error comes,
raise Exception("Unexpected return value from NetMHCstabpan server. Unable to parse response.\n{}".format(response.content.decode())) Exception: Unexpected return value from NetMHCstabpan server. Unable to parse response.

[asmlgkj]

when try to get omething about chop, error happeds too
docker run --rm --user id -u:id -g -v $4:/data docker.io/griffithlab/pvactools pvacseq run /data/$3 $1 ${content_hla} MHCnuggetsI /data/pvacseq -e1 8,9,10,11 --iedb-install-directory /opt/iedb -t 4 --tdna-vaf 0.01 --normal-sample-name $1_N -k --downstream-sequence-length 1000 -c 1 --exclude-NAs --net-chop-method cterm

[susannasiebert]

Is this from the same input VCF as #722?

The error you are seeing with NetChop (No address associated with hostname) looks to me like maybe their server was down for a bit. I can't reproduce this error on my end. Can you try running with just --net-chop-method cterm again and see if you are still seeing this error or a a different one?

[asmlgkj]

thanks a lot, it is the same vcf input, do you know how often is the server down

[susannasiebert]

I have only seen it down once before but I'm also not running their tool consistently.

[asmlgkj]

thanks a lot, --netmhc-stab can not be used until now
sometimes I found -t 4 can be much faster than -t 16, the -t 16 always can not get final result after a long time, I do not know whether it is like this

[rbowenj]

I am also encountering this issue. Roughly half the pVACtools jobs I run error out with this NetMHCStabPan exception, but the other half don't. Usually rerunning a failed job overcomes the issue.

[susannasiebert]

Unexpected return value from NetMHCstabpan server. Unable to parse response. is an error thrown by pVACtools directly when NetMHCstabpan returns html content that is unexpected and we have no logic to handle that content. I will need to know the full html output that is being emitted after that message in order to determine the problem with NetMHCstabpan.

@rbowenj and @asmlgkj can you please post the full html body of the NetMHCstabpan error message you are seeing?

[asmlgkj]

@susannasiebert thanks a lot. it is the output of the vcf I sent to you before, my server is in full run and has no mem for it, I can run it tomorrow

[susannasiebert]

Do you remember in the screenshot from this comment whether that run had the same error message with "Cannot find file /0.pep"? I'm currently just trying to figure out if NetMHCstabpan is returning any other errors aside from this specific one.

[asmlgkj]

thanks a lot. I really can not rememeber this, sorry for that. I will try to build the environment in another server to give you feedback if time is ok. I am running it now

[susannasiebert]

No worries. I will also run it on my end to try and replicate this error.

[susannasiebert]

I've been able to replicate this error only once, the first time running your example command, but not again, even when running the exact same command a second time or when trying different ways to run the same command (standalone, docker -it, etc). This leads me to believe that this error must be intermittent. I will make a bugfix to add a retry to the NetMHCstabpan command when this error is encountered in the hopes that retrying will resolve this issue.

[asmlgkj]

thanks a lot. here is a rerun. all the log is here, hope it can help the fix.
Running NetMHCStabPan /usr/local/lib/python3.7/site-packages/lib/aggregate_all_epitopes.py:175: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only df.drop('rank_tier', 1, inplace=True) /usr/local/lib/python3.7/site-packages/lib/aggregate_all_epitopes.py:176: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only df.drop('rank_ic50', 1, inplace=True) /usr/local/lib/python3.7/site-packages/lib/aggregate_all_epitopes.py:177: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only df.drop('expr', 1, inplace=True) /usr/local/lib/python3.7/site-packages/lib/aggregate_all_epitopes.py:178: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only df.drop('rank_expr', 1, inplace=True) /usr/local/lib/python3.7/site-packages/lib/aggregate_all_epitopes.py:179: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only df.drop('rank', 1, inplace=True) Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 73, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.7/socket.py", line 752, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 353, in connect conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7efe27d4fb50>: Failed to establish a new connection: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='services.healthtech.dtu.dk', port=443): Max retries exceeded with url: /cgi-bin/webface2.cgi?jobid=61775EC100005029DCDB1380&wait=20 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efe27d4fb50>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/pvacseq", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/tools/pvacseq/main.py", line 95, in main args[0].func.main(args[1]) File "/usr/local/lib/python3.7/site-packages/tools/pvacseq/run.py", line 122, in main pipeline.execute() File "/usr/local/lib/python3.7/site-packages/lib/pipeline.py", line 504, in execute PostProcessor(post_processing_params).execute() File "/usr/local/lib/python3.7/site-packages/lib/post_processor.py", line 36, in execute self.call_netmhc_stab() File "/usr/local/lib/python3.7/site-packages/lib/post_processor.py", line 125, in call_netmhc_stab self.netmhc_stab_fh.name, File "/usr/local/lib/python3.7/site-packages/lib/netmhc_stab.py", line 61, in main response = query_netmhcstabpan_server(staging_file, peptide_lengths, allele_list, jobid_searcher) File "/usr/local/lib/python3.7/site-packages/lib/netmhc_stab.py", line 123, in query_netmhcstabpan_server response = requests.get(response.url) File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='services.healthtech.dtu.dk', port=443): Max retries exceeded with url: /cgi-bin/webface2.cgi?jobid=61775EC100005029DCDB1380&wait=20 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efe27d4fb50>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

[asmlgkj]

but if go to the https://services.healthtech.dtu.dk/service.php?NetMHCstabpan-1.0. the example data can run normally

[susannasiebert]

ok, so we still have two different errors going on: Failed to establish a new connection: [Errno -5] No address associated with hostname and Cannot open file <path>/0.pep>. Do you consistently get these errors? Do the calls to NetMHCstabpan ever succeed?

[asmlgkj]

thanks a lot yes, continous, never succeed, and I hope a consistent success

[susannasiebert]

This sounds like a problem with your network setup to me. Can you try running the following: docker run --rm docker.io/griffithlab/pvactools curl https://services.healthtech.dtu.dk/cgi-bin/webface2.cgi

[asmlgkj]

docker run --rm docker.io/griffithlab/pvactools curl https://services.healthtech.dtu.dk/cgi-bin/webface2.cgi

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0curl: (6) Could not resolve host: services.healthtech.dtu.dk
does this mean it is a problem of my net. this is the machine I am using thanks a lot

[asmlgkj]

$docker run --rm docker.io/griffithlab/pvactools curl https://services.healthtech.dtu.dk/cgi-bin/webface2.cgi % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 273 0 273 0 0 114 0 --:--:-- 0:00:02 --:--:-- 114

Exception:	WebfaceConfigError
Package:	Webface : 401
Message:	Jobid not provided

[susannasiebert]

So one time when you executed that command it return a "Could not resolve host: services.healthtech.dtu.dk" and the other time it returned the 401 error (which is what is expected if it works)?

[asmlgkj]

the Could not resolve host: services.healthtech.dtu.dk is the server I am suing now and the error post all comes from this sever
the 401 is aother server

[susannasiebert]

ok, so you won't be able to run it on the server that gets that error. You should contact your IT department to see if there is some configuration that needs changing to get it to work unless they restricted internet access for that server on purpose.

The server that gets the 401 can be used.

[asmlgkj]

Thanks a lot, is there any suggestion for this?

[susannasiebert]

Unfortunately, since are not familiar with your IT system, you will need to talk to your system administrator or IT department.

[susannasiebert]

NetMHCstabpan and NetChop are also optional so you can always run pVACseq without them.

[asmlgkj]

thanks a lot I want to get these results if possible, and has error 'Cannot open file /0.pep>' been solved.
do you have time to have a look at https://github.com/griffithlab/pVACtools/issues/726

[susannasiebert]

Unfortunately, I'm currently primarily working on a different project so I don't have a lot of time to work on pVACtools. I'm working on these bugs as my time allows and there are quite a few open issues that all need addressing. I'm currently working on the "'Cannot open file /0.pep>'" error and will update this issue once I make a pull request for it.

[asmlgkj]

thanks a lot but directly ping is ok

[susannasiebert]

I don't know what else to tell you. It's obviously not working on that server when you actually try to run it. It's not a problem with pVACtools but with your network. There is nothing I can do on our end to resolve this for you. Please talk to your IT department to figure out why it's not working.

[asmlgkj]

now this serve can run docker run --rm docker.io/griffithlab/pvactools curl https://services.healthtech.dtu.dk/cgi-bin/webface2.cgi. I also get in the docker ,also ok and return 401, does it mean dtu server is not stable?

[susannasiebert]

That address is what the form gets submitted to so when only running curl no data is submitted and it throws a 401 error. That is not a problem when we run it with pVACtools.

[asmlgkj]

Thanks a lot. I mean sometimes can not connect, sometimes return 401(maybe lack some parameters, but connect is ok, does it mean dtu server is not stable?

[susannasiebert]

I don't know.

[asmlgkj]

Thanks a lot.

[rbowenj]

@susannasiebert thanks for looking into this. Not sure how helpful this is on top of what's already been said, but I'll put it here in case you need more info. To me it seems like the simple NetMHCStabPan retry you mentioned will help get around this issue.

I get the following error intermittently, sometimes from pVACseq and sometimes from pVACfuse. Only about 1/4 of my jobs fail with this error - other jobs (running simultaneously with the same command but different input files) have no problem getting NetMHCStabPan results.

The complete error is:

2021-10-25T05:55:17.917534631Z Traceback (most recent call last):
2021-10-25T05:55:17.917553275Z   File "/usr/local/bin/pvacfuse", line 8, in <module>
2021-10-25T05:55:17.917558878Z     sys.exit(main())
2021-10-25T05:55:17.917563453Z   File "/usr/local/lib/python3.7/site-packages/tools/pvacfuse/main.py", line 73, in main
2021-10-25T05:55:17.917568022Z     args[0].func.main(args[1])
2021-10-25T05:55:17.917571986Z   File "/usr/local/lib/python3.7/site-packages/tools/pvacfuse/run.py", line 108, in main
2021-10-25T05:55:17.917575072Z     pipeline.execute()
2021-10-25T05:55:17.917577822Z   File "/usr/local/lib/python3.7/site-packages/lib/pipeline.py", line 504, in execute
2021-10-25T05:55:17.917580790Z     PostProcessor(**post_processing_params).execute()
2021-10-25T05:55:17.917583639Z   File "/usr/local/lib/python3.7/site-packages/lib/post_processor.py", line 36, in execute
2021-10-25T05:55:17.917586522Z     self.call_netmhc_stab()
2021-10-25T05:55:17.917589186Z   File "/usr/local/lib/python3.7/site-packages/lib/post_processor.py", line 125, in call_netmhc_stab
2021-10-25T05:55:17.917592122Z     self.netmhc_stab_fh.name,
2021-10-25T05:55:17.917594793Z   File "/usr/local/lib/python3.7/site-packages/lib/netmhc_stab.py", line 93, in main
2021-10-25T05:55:17.917598433Z     raise Exception("Unexpected return value from NetMHCstabpan server. Unable to parse response.\n{}".format(response.content.decode()))
2021-10-25T05:55:17.917601834Z Exception: Unexpected return value from NetMHCstabpan server. Unable to parse response.
2021-10-25T05:55:17.917604822Z <BODY LINK="#0000ff" BGCOLOR="#ffffff">
2021-10-25T05:55:17.917615957Z 
2021-10-25T05:55:17.917618769Z <script language=JavaScript type = "text/javascript">
2021-10-25T05:55:17.917621687Z   var loc = window.top.location.search.substring(1);
2021-10-25T05:55:17.917624448Z   if (loc != null && loc !== "" && loc.length > 1) {
2021-10-25T05:55:17.917627619Z       document.write('<center><h2>',loc,' Server Output - DTU Health Tech</h2></center>');}
2021-10-25T05:55:17.917631639Z   </script> 
2021-10-25T05:55:17.917635401Z <hr>
2021-10-25T05:55:17.917639958Z <pre>
2021-10-25T05:55:17.917643918Z 
2021-10-25T05:55:17.917648052Z # NetMHCstabpan version 1.0
2021-10-25T05:55:17.917652050Z 
2021-10-25T05:55:17.917654810Z # Input is in FSA format
2021-10-25T05:55:17.917657616Z 
2021-10-25T05:55:17.917660234Z # Peptide length 10,9,11
2021-10-25T05:55:17.917662960Z 
2021-10-25T05:55:17.917665701Z HLA-A23:01 : Distance to traning data  0.000 (using nearest neighbor HLA-A23:01)
2021-10-25T05:55:17.917668534Z 
2021-10-25T05:55:17.917671169Z Cannot open file /var/www/webface/tmp/server/netMHCstabpan/617646BA000037662A0F662D/3779/0.pep
2021-10-25T05:55:17.917674123Z </pre>
2021-10-25T05:55:17.917676983Z <hr>
2021-10-25T05:55:17.917679853Z Go <a href="javascript:history.back()"><b>back</b></a>.
2021-10-25T05:55:17.917683083Z <!--
2021-10-25T05:55:17.917685883Z <button type="button"
2021-10-25T05:55:17.917688709Z onclick="window.top.location.reload(true);")>
2021-10-25T05:55:17.917691667Z New Submission</button>
2021-10-25T05:55:17.917694586Z -->
2021-10-25T05:55:17.917697275Z </BODY>
2021-10-25T05:55:17.917699979Z </HTML>
2021-10-25T05:55:17.917702730Z 
2021-10-25T05:55:17.917705404Z 

[asmlgkj]

Thanks a lot this time it runs successful. I think there must be something something with the dtu server. but I think the result is hard to select
the allele is whih to use for the neoantigen of Stability of the pMHC-I complex

[susannasiebert]

@asmlgkj This is expected. NetMHCstabpan doesn't support all possible alleles so in cases where pVACseq submits an unsupported allele NetMHCstabpab will use the "nearest neighbor". The allele used internally by NetMHCstabpan is recorded in the "NetMHCstab allele" column.

[susannasiebert]

@rbowenj Thank you for submitting the full error log. I'm currently working on implementing a retry strategy for this error.

[asmlgkj]

thanks a lot, but in the site of NetMHCstabpan. it seems has HLA-C*03:03，why it use the nearest here? <>

[susannasiebert]

If you actually run a peptide with that HLA you will see the following return:

Why that is, you would need to contact the developers of NetMHCstabpan. My assumption is that they didn't train on that allele directly. Table 1 in their paper makes it look like they only trained on HLA-A and HLA-B alleles.

[asmlgkj]

thanks a lot . sorry for not further investigation

[susannasiebert]

2.0.5 has just been released, which adds a retry when the Cannot open file error is encountered. Please switch to this version and see if it resolves your problem. I will close this issue for now but please do reopen it if you're still encountering this error after switching to 2.0.5.