Closed rp87701n closed 4 years ago
I actually have the same problem with melbourne backend since yesterday's evening (CEST). I cannot send ANY job because I get the error about too many concurrent job requests, even though I do not have any jobs pending (according to my profile's dashboard). 5-qubit backends work normally and I can send jobs to them.
On the public devices, including ibmq_qasm_simulator
, you can only have 5 unfinished jobs at a time.
@fbm2718 Are you saying that you verified you had no pending jobs before trying to run a job, yet the job submit still failed with the "maximum number reached" error?
@rp87701n The issue should be fixed now, so can you try running jobs again? Don't forget there's a limit of 5 unfinished jobs. If it somehow still doesn't work, can you post the failing job ID and the backend you used? Thanks!
I just tried with ibmq_london instance and got the same error. please find job id 5e8d45f8db35e20012d27877 5e8d45f5628fb5001299d32a 5e8d45f18de4b40012aa08dc 5e8d45eb628fb5001299d328 5e8d45e78de4b40012aa08da
got the same error here is the job submitted. 5e8d4933db35e20012d27893 JobStatus.QUEUED 5e8d492fc2693200125d4568 JobStatus.QUEUED 5e8d492cc9b49f00110be22f JobStatus.QUEUED 5e8d492a9b8c770011088162 JobStatus.QUEUED 5e8d4926c2693200125d4566 JobStatus.RUNNING
here is the job id for the same. 5e8d49f89b8c770011088169 JobStatus.QUEUED 5e8d49f4c2693200125d4571 JobStatus.QUEUED 5e8d49f1fa2e94001232aede JobStatus.QUEUED 5e8d49eec2693200125d4570 JobStatus.QUEUED 5e8d49e7158d5e001135c13c JobStatus.QUEUED
Note: I run this test on each device separately to see if any of this device is working or not. at each time when I am running test, I make sure that if any jobs pending or running and cancel them before running.
here is a snippet of the code that I am running and error that I got during this test.
tried with Backend: 'ibmqx2 ... got the same error here is the job submitted. 5e8d4933db35e20012d27893 JobStatus.QUEUED 5e8d492fc2693200125d4568 JobStatus.QUEUED 5e8d492cc9b49f00110be22f JobStatus.QUEUED 5e8d492a9b8c770011088162 JobStatus.QUEUED 5e8d4926c2693200125d4566 JobStatus.RUNNING
checke with Backend: 'ibmq_16_melbourne here is the job id for the same. 5e8d49f89b8c770011088169 JobStatus.QUEUED 5e8d49f4c2693200125d4571 JobStatus.QUEUED 5e8d49f1fa2e94001232aede JobStatus.QUEUED 5e8d49eec2693200125d4570 JobStatus.QUEUED 5e8d49e7158d5e001135c13c JobStatus.QUEUED
@rp87701n In both cases you described above, it seems that you submitted 5 jobs to each device, all 5 jobs were accepted (hence statuses were QUEUED/RUNNING). But the 6th job to each device was rejected? If so, that's the expected behavior as you can only have 5 unfinished jobs at a time. A job is "unfinished" if it's in one of INITIALIZING, QUEUED, VALIDATING, and RUNNING states.
@jyu00 No, it is not rejected all jobs were completed. as I explained earlier.
@rp87701n Due to backend limitation on how many circuits are allowed in a job, qiskit-aqua may split an experiment into multiple jobs. When I ran your test case, I could see it tried to submit at least 6 jobs. The first 5 were submitted successfully (in either QUEUED or RUNNING state on my dashboard), but the 6th one failed with the same "job limit reached" error.
Having said that, I can create a patch so that if the job limit is reached, it'll wait for the earlier jobs to finish before submitting a new one, rather than failing the experiment.
thanks and understood. Not sure why it is submitting 6 jobs. can I run one job at a time? or is there a way that we can control no. of jobs [any parameter can control a number of jobs that users can run]. Say concurrent_job=1 then it should submit only one job. so that user can control job scheduling
Currently there is no option to specify the number of concurrent jobs, but I can add that in when I do the patch.
thanks that would help us. Do you know when this patch get released for testing.
@rp87701n I'm hoping to have it available sometime next week.
@fbm2718 Are you still having issues with ibmq_16_melbourne
? If so, can you post a test case?
@jyu00 Hey, sorry for not reaching out earlier. I was kind of in a hurry and found an inelegant workaround - I simply used different account and using it I can send job requests to melbourne without any problems. (I also tried resetting token, this did not help)
I just ran the test case on the faulty account:
Here I delete and enable account once again to make sure I am using proper, faulty account's token. Note that I send only single job.
This is what I get as the output:
`/home/fbm/anaconda3/envs/GWE/bin/python /home/fbm/0_Research/Projects/test_case_ibm/test_case_ibm.py
Traceback (most recent call last):
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/api/session.py", line 207, in request
response.raise_for_status()
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/ibmqbackend.py", line 172, in _submit_job
job_tags=job_tags)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/api/clients/account.py", line 200, in job_submit
job_tags=job_tags)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/api/clients/account.py", line 232, in _job_submit_post
job_tags=job_tags)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/api/rest/root.py", line 143, in job_submit
return self.session.post(url, json=payload).json()
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/api/session.py", line 226, in request
raise RequestsApiError(message) from ex
qiskit.providers.ibmq.api.exceptions.RequestsApiError: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/fbm/0_Research/Projects/test_case_ibm/test_case_ibm.py", line 25, in <module>
qobj_id = 'test_case')
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/execute.py", line 245, in execute
return backend.run(qobj, **run_config)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/ibmqbackend.py", line 132, in run
return self._submit_job(qobj, job_name, api_job_share_level, job_tags)
File "/home/fbm/anaconda3/envs/GWE/lib/python3.7/site-packages/qiskit/providers/ibmq/ibmqbackend.py", line 174, in _submit_job
raise IBMQBackendApiError('Error submitting job: {}'.format(str(ex)))
qiskit.providers.ibmq.exceptions.IBMQBackendApiError: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'"
Process finished with exit code 1
And here's a screenshot of my current dashboard:
If I run the very same code but with a token from a different account, then it works fine.
Thanks a lot!
P.S.: I think that patch changing the behaviour under submitting more than the allowed number of jobs is good idea, so far I've been sending more jobs using a loop with check if qiskit still returns an error. ;-)
@fbm2718 Your old account should be fixed now if you want to use it. Let me know if you encounter this issue again.
@jyu00 Okay, I've just tried it and it works on the old account. Thanks a lot! (out of curiosity - was that very complicated bug?)
I am trying to run the qsvm on simulator and it is taking lot of time and want to reduce training time. what are hyperparameter that i can use it . do you know where 1000 came from? can I control it?
here is the log
2020-04-15 21:55:43,636:qiskit.aqua.components.feature_maps.pauli_expansion:WARNING: Due to the limited entangler_map, ZIIZ is skipped. 2020-04-15 21:55:43,637:qiskit.aqua.components.feature_maps.pauli_expansion:WARNING: Due to the limited entangler_map, ZIZI is skipped. 2020-04-15 21:55:43,638:qiskit.aqua.components.feature_maps.pauli_expansion:INFO: Pauli terms include: ['IIIZ', 'IIZI', 'IZII', 'ZIII', 'IIZZ', 'IZZI', 'ZZII'] 2020-04-15 21:55:43,639:qiskit.aqua.quantum_instance:INFO: Qiskit Terra version: 0.12.0 Backend: 'qasm_simulator (AerProvider)', with following setting: {'basis_gates': ['u1', 'u2', 'u3', 'cx', 'cz', 'id', 'x', 'y', 'z', 'h', 's', 'sdg', 't', 'tdg', 'swap', 'ccx', 'unitary', 'initialize', 'cu1', 'cu2', 'cu3', 'cswap', 'mcx', 'mcy', 'mcz', 'mcu1', 'mcu2', 'mcu3', 'mcswap', 'multiplexer', 'kraus', 'roerror'], 'coupling_map': None} {'pass_manager': None, 'initial_layout': None, 'seed_transpiler': 10598, 'optimization_level': None} RunConfig(max_credits=10, seed_simulator=10598, shots=1024) {'timeout': None} {} {} Measurement mitigation: None 2020-04-15 21:56:38,442:qiskit.aqua.algorithms.many_sample.qsvm.qsvm:DEBUG: Calculating overlap: |██████████████████████████████████████████████████| 1000/1000 [00:00:00:00] 2020-04-15 21:58:39,465:qiskit.aqua.algorithms.many_sample.qsvm.qsvm:DEBUG: Calculating overlap: |██████████████████████████████████████████████████| 1000/1000 [00:00:00:00] 2020-04-15 22:00:38,331:qiskit.aqua.algorithms.many_sample.qsvm.qsvm:DEBUG: Calculating overlap: |--------------------------------------------------| 0/1000 []
@jyu00 Okay, I've just tried it and it works on the old account. Thanks a lot! (out of curiosity - was that very complicated bug?)
@fbm2718 I don't actually what the bug was. I contacted Quantum Experience support and they fixed it lol.
@rp87701n I don't know the answer to that. You'll likely get an answer faster if you ask on Slack or Stack Exchange.
ok, thanks, I will check in slack . can we get a patch released this week.
I tested code again today and getting the same error.
HTTPError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/qiskit/providers/ibmq/api/session.py in request(self, method, url, bare, kwargs) 229 response = super().request(method, final_url, kwargs) --> 230 response.raise_for_status() 231 except RequestException as ex:
/opt/conda/lib/python3.7/site-packages/requests/models.py in raise_for_status(self) 940 if http_error_msg: --> 941 raise HTTPError(http_error_msg, response=self) 942
HTTPError: 400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs
The above exception was the direct cause of the following exception:
RequestsApiError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/qiskit/providers/ibmq/ibmqbackend.py in _submit_job(self, qobj, job_name, job_share_level, job_tags) 213 job_share_level=job_share_level, --> 214 job_tags=job_tags) 215 except ApiError as ex:
/opt/conda/lib/python3.7/site-packages/qiskit/providers/ibmq/api/clients/account.py in job_submit(self, backend_name, qobj_dict, job_name, job_share_level, job_tags) 181 job_share_level=_job_share_level, --> 182 job_tags=job_tags) 183
/opt/conda/lib/python3.7/site-packages/qiskit/providers/ibmq/api/rest/root.py in create_remote_job(self, backend_name, job_name, job_share_level, job_tags) 155 --> 156 return self.session.post(url, json=payload).json() 157
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, kwargs) 577 --> 578 return self.request('POST', url, data=data, json=json, kwargs) 579
/opt/conda/lib/python3.7/site-packages/qiskit/providers/ibmq/api/session.py in request(self, method, url, bare, **kwargs) 248 --> 249 raise RequestsApiError(message) from ex 250
RequestsApiError: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'
The above exception was the direct cause of the following exception:
IBMQBackendApiError Traceback (most recent call last)
Qiskit Software | Version |
---|---|
Qiskit | 0.18.1 |
Terra | 0.13.0 |
Aer | 0.5.0 |
Ignis | 0.3.0 |
Aqua | 0.6.6 |
IBM Q Provider | 0.6.0 |
Python | 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20) [GCC 7.3.0] OS | Linux CPUs | 8 Memory (Gb) | 31.420135498046875 Thu Apr 23 01:41:34 2020 UTC
i upgraded both qiskit and Terra version
Hi, the fix has taken place in the master branch of the repository here and is not yet included in an official release. The plan is for a release very shortly - until then, to try it, you would have to clone the repositories here - aqua, terra etc and install from the source files. Here is a link that describes this if you are interested https://qiskit.org/documentation/contributing_to_qiskit.html#installing-from-source
thanks for the update. I upgraded the new packages. it seems my old code is not working. i guess something changed with this version as well.
Trying to run the same code that was worked earlier but not it is not working.
aqua_globals.random_seed = 1024 feature_map = SecondOrderExpansion(feature_dimension=get_feature_dimension(training_input), depth=2, entanglement='linear') extensions = [ OneAgainstRest(_QSVM_Estimator, [feature_map])] # this line throwing error do i need to create MulticlassExtension(_QSVM_Estimator) like this. is there any code snipet for this new changes.
for extension in extensions: svm = QSVM(feature_map, training_input, test_input, total_array,multiclass_extension=extension) quantum_instance = QuantumInstance(backend, shots=1024, seed_simulator=aqua_globals.random_seed, seed_transpiler=aqua_globals.random_seed, circuit_caching=False, skip_qobj_validation=False,job_callback=job_callback) result = svm.run(quantum_instance) for k,v in result.items(): print("'{}' : {}".format(k, v))
TypeError Traceback (most recent call last)
Hi, you can take a look at this test case https://github.com/Qiskit/qiskit-aqua/blob/master/test/aqua/test_qsvm.py which shows what I think you need
thanks . I can see the applied change working now. but after 30 to 45 mins later getting a qiskit.providers.ibmq.job.exceptions.IBMQJobTimeoutError: 'Timeout while waiting for job 5ea667291070d7001959644a.' error.
here is fix for the earlier issues which is worked. 2020-04-27 05:34:02,449:qiskit.aqua.components.multiclass_extensions.all_pairs:INFO: Require 45.0 estimators. 2020-04-27 05:34:14,523:qiskit.aqua.utils.run_circuits:WARNING: FAILURE: Can not get job id, Resubmit the qobj to get job id. Terra job error: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'" 2020-04-27 05:34:14,890:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea66ed0a100c50018f77078 to finish before submitting the next one. ^CTraceback (most recent call last): File "/app/anaconda3/envs/QiskitDevenv/lib/python3.6/site-packages/qiskit/providers/ibmq/api/session.py", line 250, in request response.raise_for_status() File "/app/anaconda3/envs/QiskitDevenv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs
later we are getting this error
Z', 'COMPLETED': '2020-04-27T05:00:07.850Z'}, _use_object_storage=True, allow_object_storage=True, cost=54.816, end_date='2020-04-27T05:00:07.608Z', hub_info={'hub': {'name': 'ibm-q'}, 'group': {'name': 'open'}, 'project': {'name': 'main'}}, ip={'ip': '3.86.182.117', 'city': 'Ashburn', 'country': 'United States', 'continent': 'North America'}, kind=<ApiJobKind.QOBJECT_STORAGE: 'q-object-external-storage'>, object_storage_info={}, properties={}, share_level='none', summary_data={'size': {'input': 63278, 'output': 14431}, 'summary': {'qobj_config': {'memory': False, 'max_credits': 10, 'type': 'QASM', 'n_qubits': 15, 'memory_slots': 2, 'shots': 1024}, 'gates_executed': 780, 'max_qubits_used': 15, 'partial_validation': False, 'num_circuits': 30}, 'success': True, 'resultTime': 26.082602901849896}, user_id='5e3b8d5eeb8baa00192a6b7c')
2020-04-27 05:01:37,600:qiskit.aqua.utils.run_circuits:WARNING: FAILURE: Can not get job id, Resubmit the qobj to get job id. Terra job error: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'"
2020-04-27 05:01:37,803:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea667291070d7001959644a to finish before submitting the next one.
Traceback (most recent call last):
File "proj.py", line 102, in
here is the stack trace for this issue.
2020-04-27 06:10:11,548:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea6773b473d6b0018016e0b to finish before submitting the next one. 2020-04-27 06:14:06,499:qiskit.aqua.utils.run_circuits:WARNING: FAILURE: Can not get job id, Resubmit the qobj to get job id. Terra job error: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'" 2020-04-27 06:14:07,206:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea6773d1070d7001959650f to finish before submitting the next one. 2020-04-27 06:16:12,185:qiskit.aqua.utils.run_circuits:WARNING: FAILURE: Can not get job id, Resubmit the qobj to get job id. Terra job error: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'" 2020-04-27 06:16:12,589:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea6773fbf76d30018086f15 to finish before submitting the next one. 2020-04-27 06:17:27,586:qiskit.aqua.utils.run_circuits:WARNING: FAILURE: Can not get job id, Resubmit the qobj to get job id. Terra job error: "Error submitting job: '400 Client Error: Bad Request for url: https://api.quantum-computing.ibm.com/api/Network/ibm-q/Groups/open/Projects/main/Jobs. Reached maximum number (5) of concurrent jobs, Error code: 3458.'" 2020-04-27 06:17:27,972:qiskit.aqua.utils.run_circuits:WARNING: Job limit reached, waiting for job 5ea6782c1070d7001959651b to finish before submitting the next one.
Traceback (most recent call last):
File "proj.py", line 102, in
Information
Python | 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20) [GCC 7.3.0] OS | Linux CPUs | 8 Memory (Gb) | 31.409027099609375 Fri Apr 03 23:22:05 2020 UTC OS: cloud-based IBM Quantum systems and simulators
What is the current behavior?
I can run the QSVM algorithm on qasm_simulator and getting the result back but when i run the same code on ibmq_16_melbourne device or any other device. getting this error
IBMQBackendApiError Traceback (most recent call last)