Put fails for 800mb file size

akrishnamo commented 7 years ago

Put command still failing....

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>dir
 Volume in drive C is Windows8_OS
 Volume Serial Number is 7A07-71C3

 Directory of C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring

30-05-2017  08:48 AM    <DIR>          .
30-05-2017  08:48 AM    <DIR>          ..
30-05-2017  08:01 AM       242,205,616 test_stg1.zip
30-05-2017  08:48 AM    <DIR>          train
30-05-2017  08:23 AM       872,167,582 train.zip
               2 File(s)  1,114,373,198 bytes
               3 Dir(s)  315,600,629,760 bytes free

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>rorocloud --verbose put train.zip /data/fish_train.zip
Starting new HTTPS connection (1): rorocloud.rorodata.com
https://rorocloud.rorodata.com:443 "POST /upload?path=/data/fish_train.zip HTTP/1.1" 502 182
Traceback (most recent call last):
  File "d:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "d:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "D:\Python27\Scripts\rorocloud.exe\__main__.py", line 9, in <module>
  File "d:\python27\lib\site-packages\rorocloud\cli.py", line 184, in main
    cli()
  File "d:\python27\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "d:\python27\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "d:\python27\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "d:\python27\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "d:\python27\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "d:\python27\lib\site-packages\rorocloud\cli.py", line 181, in put
    return client.put_file(source, target)
  File "d:\python27\lib\site-packages\rorocloud\client.py", line 137, in put_file
    return self._request("POST", "/upload?path="+target, files=files)
  File "d:\python27\lib\site-packages\rorocloud\client.py", line 87, in _request
    return response.json()
  File "d:\python27\lib\site-packages\requests\models.py", line 866, in json
    return complexjson.loads(self.text, **kwargs)
  File "d:\python27\lib\site-packages\simplejson\__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "d:\python27\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "d:\python27\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>

akrishnamo commented 7 years ago

trying to reproduce....not able to connect to rorocloud

rorocloud_put

smaller files seem to go through

sohail535 commented 7 years ago

nabarun pal is able to connect to rorocloud. @akrishnamo are you still not able to connect?

akrishnamo commented 7 years ago

broke again on retry...seems like a systemic issue...

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>dir
 Volume in drive C is Windows8_OS
 Volume Serial Number is 7A07-71C3

 Directory of C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring

30-05-2017  10:50 AM    <DIR>          .
30-05-2017  10:50 AM    <DIR>          ..
30-05-2017  10:49 AM             2,849 errors.txt
30-05-2017  08:01 AM       242,205,616 test_stg1.zip
30-05-2017  08:48 AM    <DIR>          train
30-05-2017  08:23 AM       872,167,582 train.zip
               3 File(s)  1,114,376,047 bytes
               3 Dir(s)  315,604,004,864 bytes free

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>rorocloud --verbose put train.zip /data/fish_train.zip
Starting new HTTPS connection (1): rorocloud.rorodata.com
https://rorocloud.rorodata.com:443 "POST /upload?path=/data/fish_train.zip HTTP/1.1" 502 182
Traceback (most recent call last):
  File "d:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "d:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "D:\Python27\Scripts\rorocloud.exe\__main__.py", line 9, in <module>
  File "d:\python27\lib\site-packages\rorocloud\cli.py", line 184, in main
    cli()
  File "d:\python27\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "d:\python27\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "d:\python27\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "d:\python27\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "d:\python27\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "d:\python27\lib\site-packages\rorocloud\cli.py", line 181, in put
    return client.put_file(source, target)
  File "d:\python27\lib\site-packages\rorocloud\client.py", line 137, in put_file
    return self._request("POST", "/upload?path="+target, files=files)
  File "d:\python27\lib\site-packages\rorocloud\client.py", line 87, in _request
    return response.json()
  File "d:\python27\lib\site-packages\requests\models.py", line 866, in json
    return complexjson.loads(self.text, **kwargs)
  File "d:\python27\lib\site-packages\simplejson\__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "d:\python27\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "d:\python27\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

C:\Users\ananth\Desktop\rorodata\Data\Kaggle\Fish_Monitoring>

akrishnamo commented 7 years ago

I don't think this is a pure storage issue. I was able to transfer the entire 850MB into /data, but in chunks. Maybe an intermediate storage / cache issue...

By the way, 850MB single file still fails (checked)

put_parts

rorodata / rorocloud

Put fails for 800mb file size #2