search

simonverret / materials_data_api_scripts

scripts to load all data from ICSD, Materials Project, and OQMD
MIT License
51 stars 14 forks source link

Error happens with icsd/download.py #3

Open YanjunLiu2 opened 2 months ago

YanjunLiu2 commented 2 months ago

Hi Simon,

When I tried to download all cifs from icsd I got the following error:

(ICSD) yanjunliu@dhcp-vl2041-23489 materials_datasets % python icsd/download.py Traceback (most recent call last): File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 153, in download_all(usrname, passwrd, ICSD_PKL) File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 131, in download_all with ICSD_Session(loginid, password) as icsd: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 25, in init self.login_token = self.login() # sets self.login_token ^^^^^^^^^^^^ File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 52, in login raise ConnectionError(login_response.headers) ConnectionError: {'Set-Cookie': 'ICSDCHECK=1727465301266; Path=/, JSESSIONID=7D1BF39A51CB269160AE2159789F9B26; Path=/; HttpOnly, FIZ-Cookie=238830221.16671.0000; path=/; Httponly; Secure', 'Content-Language': 'en', 'Content-Type': 'text/plain', 'Content-Length': '104', 'Date': 'Fri, 27 Sep 2024 19:28:20 GMT', 'Keep-Alive': 'timeout=20', 'Connection': 'keep-alive', 'Strict-Transport-Security': 'max-age=16070400; includeSubDomains'}

Could you take a look at what the reason is? Thank you!

Best wishes,

Yanjun

simonverret commented 2 months ago

Dear Yanjun,

Thanks for trying the script out. It seems to be a login error. You need proper (paid) credentials to connect to ICSD.

I don't have credential anymore myself, so I cannot troubleshoot the error. Sorry.

YanjunLiu2 commented 1 month ago

Dear Simon,

Thanks for your reply! To confirm, the credentials json file should be placed in the icsd folder where the download.py is, right? Does the attached file look correct? Because I can use my paid account to access their api in https://icsd.fiz-karlsruhe.de/api/#/authentication/doLogin. So this problem is kind of unexpected icsd_credentials.json

YanjunLiu2 commented 1 month ago

Hi Simon,

I ran the program again, and it started running normally, but then it encountered an error in the middle：

(ICSD) yanjunliu@dhcp-vl2041-51928 materials_datasets % python icsd/download.py logged in ICSD (token=D810CE706E80F95A00FA45BE46CB39F6) materials with 1 elements received 3684 ids querying cifs reconnect 1/4 token:3E53939BF22C4A141DE49A6F42B6C864 reconnect 2/4 token:249882D15FAD1643BD5432896A359740 reconnect 3/4 token:ED21B2A1268C34D421D3B01BF71F0674 reconnect 4/4 token:36B190E0D0F49FA3947481C7EC542A62 received 3684/3684 cif strings materials with 2 elements received 54441 ids querying cifs reconnect 1/55 token:CD9D925F030A8984DA624D6834F37615 reconnect 2/55 token:253F035B7B6F8F890339D7C648E4C4C3 reconnect 3/55 token:13AE5CD702B40A591FBA407268BDD49A reconnect 4/55 token:1CB7D04ADCD50639EEC263215759804B reconnect 5/55 token:A9E4F21C68645A008623DF451048EC40 reconnect 6/55 token:0024C075308F75610902B2C06A952D6F reconnect 7/55 token:01D4D75EAF426EF483642E66FAA86548 reconnect 8/55 token:D1CD73C58B163159561F4F0C698B1311 reconnect 9/55 token:2840CEF167D08AF9CDA48BDE81DE30AE reconnect 10/55 token:E18996A00B3D068DDCC0DD09995A4B95 reconnect 11/55 token:0E7CE1B92D2006C0919B039C7D04E6E1 reconnect 12/55 token:F04295D79EBBA72B23B4FB2ADF9B34E2 reconnect 13/55 token:F77012EF841CAE9F531318E5BA44DF07 reconnect 14/55 token:6D2E762B09768B81356D24056FCC53F0 reconnect 15/55 token:0E0968450630358410BE6F0B5E4EEC84 reconnect 16/55 token:B576294BD92DE0A3022B81CB6ABEAFC9 reconnect 17/55 token:8C0AF19DE91848B46B80F0F277B0AD5D reconnect 18/55 token:832FE3CC3960D5596E78E085F8F7AAE6 reconnect 19/55 token:9C3F6035CC7B34530F3B01AF6E059ACD reconnect 20/55 token:6E9C09B79F1A62D8BFA6F0B8AF37FDDC logged out ICSD (token=6E9C09B79F1A62D8BFA6F0B8AF37FDDC) Traceback (most recent call last): File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 748, in _error_catcher yield File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 1209, in read_chunked chunk = self._handle_chunk(amt) ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 1155, in _handle_chunk returned_chunk = self._fp._safe_read(self.chunk_left) # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/http/client.py", line 642, in _safe_read raise IncompleteRead(data, amt-len(data)) http.client.IncompleteRead: IncompleteRead(5218 bytes read, 2974 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/models.py", line 820, in generate yield from self.raw.stream(chunk_size, decode_content=True) File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 1057, in stream yield from self.read_chunked(amt, decode_content=decode_content) File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 1189, in read_chunked with self._error_catcher(): File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/contextlib.py", line 158, in exit self.gen.throw(value) File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/urllib3/response.py", line 775, in _error_catcher raise ProtocolError(f"Connection broken: {e!r}", e) from e urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(5218 bytes read, 2974 more expected)', IncompleteRead(5218 bytes read, 2974 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 153, in download_all(usrname, passwrd, ICSD_PKL) File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 139, in download_all cif_list.extend(icsd.safe_query_cifs(id_list)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 122, in safe_query_cifs list_of_cifs.extend(self.query_cifs(to_query)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 106, in query_cifs query_results = self.raw_query(query_string, query_headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/Documents/Read_database/materials_data_api_scripts-master/materials_datasets/icsd/download.py", line 78, in raw_query query_response = self.session.get( ^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/sessions.py", line 746, in send r.content File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/models.py", line 902, in content self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b"" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/yanjunliu/opt/anaconda3/envs/ICSD/lib/python3.12/site-packages/requests/models.py", line 822, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(5218 bytes read, 2974 more expected)', IncompleteRead(5218 bytes read, 2974 more expected))

Can you please take a look and see what's causing the error? Thank you!

Best wishes,

Yanjun