ORNL / DataFed

A Federated Scientific Data Management System
https://ornl.github.io/DataFed/
Other
18 stars 14 forks source link

Challenges in uploading download data from Windows machine using Client #586

Closed ssomnath closed 3 years ago

ssomnath commented 3 years ago

I am working from my Windows 10 machine with running datafed 1.1.0:0 from pypi and am using Python 3.6

I can create records, view collections, edit items just fine using the CLI and CommandLib. However, uploads and downloads fail if I do not specify the full path of the file including the Endpoint's UUID or Legacy name. These commands work just fine on Linux and Mac but not on Windows. I am suspecting some kind of a path manipulation issue in MessageLib. Here are some experiments I tried on the CLI:

CLI:

PS C:\Users\acesu\Documents> datafed data put -w d/35440412 ./low_speeds.PNG Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed data put -w d/35440412 C:\Users\acesu\Documen ts\low_speeds.PNG Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed data put d/35440412 C:\Users\acesu\Documents\ low_speeds.PNG Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed data put d/35440412 C:/Users/acesu/Documents/ low_speeds.PNG Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed data put d/35440412 u_r4zf3725mzhidlsynbnc73e swy#226ac6d8-5e88-11eb-a46d-0e095b4c2e55/C/Users/acesu/Documents/low_speeds.PNG Task ID: task/35443097 Type: Data Put Status: Ready Started: 01/24/2021,19:15 Updated: 01/24/2021,19:15 PS C:\Users\acesu\Documents> datafed data put d/35440412 226ac6d8-5e88-11eb-a46d-0 e095b4c2e55/C/Users/acesu/Documents/low_speeds.PNG Task ID: task/35443194 Type: Data Put Status: Ready Started: 01/24/2021,19:16 Updated: 01/24/2021,19:16

PS C:\Users\acesu\Documents> datafed ep get 226ac6d8-5e88-11eb-a46d-0e095b4c2e55

PS C:\Users\acesu\Documents> datafed data put d/35440412 C/Users/acesu/Documents/l ow_speeds.PNG Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed ep default set u_r4zf3725mzhidlsynbnc73eswy#2 26ac6d8-5e88-11eb-a46d-0e095b4c2e55 u_r4zf3725mzhidlsynbnc73eswy#226ac6d8-5e88-11eb-a46d-0e095b4c2e55

PS C:\Users\acesu\Documents> datafed data put d/35440412 C/Users/acesu/Documents/l ow_speeds.PNG Unexpected token U in JSON at position 97 (source: dbPost:261 code:1)

PS C:\Users\acesu\Documents> datafed ep get u_r4zf3725mzhidlsynbnc73eswy#226ac6d8-5e88-11eb-a46d-0e095b4c2e55

Python CommandLib

dataPut

df_api.dataPut('d/35440412', './parameters.json', wait=True, # Waits until transfer completes. )

Exception Traceback (most recent call last)

in 1 put_resp = df_api.dataPut(record_id, 2 './parameters.json', ----> 3 wait=True, # Waits until transfer completes. 4 ) 5 print(put_resp) ~\Anaconda3\lib\site-packages\datafed\CommandLib.py in dataPut(self, data_id, path, encrypt, wait, timeout_sec, extension, context) 494 msg.ext = extension 495 --> 496 reply = self._mapi.sendRecv( msg ) 497 498 if ( reply[0].HasField( "task" ) == True ) and wait: ~\Anaconda3\lib\site-packages\datafed\MessageLib.py in sendRecv(self, msg, timeout, nack_except) 299 self.send( msg ) 300 _timeout = (timeout if timeout != None else self._timeout) --> 301 reply, mt, ctxt = self.recv( _timeout, nack_except ) 302 if reply == None: 303 return None, None ~\Anaconda3\lib\site-packages\datafed\MessageLib.py in recv(self, timeout, nack_except) 343 if msg_type == "NackReply" and _nack_except: 344 if reply.err_msg: --> 345 raise Exception(reply.err_msg) 346 else: 347 raise Exception("Server error {}".format( reply.err_code )) Exception: Unexpected token U in JSON at position 68 (source: dbPost:261 code:1) ### dataGet ``df_api.dataGet(['d/35437908'], os.path.abspath('.') , wait=True)`` Exception Traceback (most recent call last) in ----> 1 dget_resp = df_api.dataGet(['d/35437908'], os.path.abspath('.')) 2 dget_resp ~\Anaconda3\lib\site-packages\datafed\CommandLib.py in dataGet(self, item_id, path, encrypt, orig_fname, wait, timeout_sec, progress_bar, context) 437 msg.orig_fname = orig_fname 438 --> 439 reply = self._mapi.sendRecv( msg ) 440 441 if reply[0].task and wait: ~\Anaconda3\lib\site-packages\datafed\MessageLib.py in sendRecv(self, msg, timeout, nack_except) 299 self.send( msg ) 300 _timeout = (timeout if timeout != None else self._timeout) --> 301 reply, mt, ctxt = self.recv( _timeout, nack_except ) 302 if reply == None: 303 return None, None ~\Anaconda3\lib\site-packages\datafed\MessageLib.py in recv(self, timeout, nack_except) 343 if msg_type == "NackReply" and _nack_except: 344 if reply.err_msg: --> 345 raise Exception(reply.err_msg) 346 else: 347 raise Exception("Server error {}".format( reply.err_code )) Exception: Unexpected token U in JSON at position 68 (source: dbPost:261 code:1)
dvstans commented 3 years ago

This has been fixed and deployed in the datafed client 1.1.0-1 (on PyPi)