search

Backblaze / B2_Command_Line_Tool

The command-line tool that gives easy access to all of the capabilities of B2 Cloud Storage
Other
542 stars 128 forks source link

Some files upload, some others won't #363

Closed shishanyu closed 7 years ago

shishanyu commented 7 years ago

Ran b2 sync in CentOS 7 and it's not performing well. Also it seems to use a lot of memory.

Using https://api.backblazeb2.com
upload 2017-08-17/accounts/ecotaxi/meta/mailserver                     
ERROR:b2.sync.action:an exception occurred in a sync action GB   14 B/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 447, in _upload_large_file
    ) for (part_index, part_range) in enumerate(part_ranges)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/deltamat.tar.gz, 2017-08-17/accounts/deltamat.tar.gz, 1502954072220): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action GB   87.4 MB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_taxiapp, 2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_taxiapp, 1502955340352): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action GB   87.4 MB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_taxitest, 2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_taxitest, 1502955360432): error("can't start new thread",) can't start new thread
upload 2017-08-17/accounts/ecotaxi/meta/homedir_paths                     
upload 2017-08-17/accounts/ecotaxi/logs/ftp.radioecotaxi.com-ftp_log.offsetftpbytes
ERROR:b2.sync.action:an exception occurred in a sync action GB   77.7 MB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/system/dirs/_var_lib_rpm.tar.gz, 2017-08-17/system/dirs/_var_lib_rpm.tar.gz, 1502953509563): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action GB   77.6 MB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 429, in _upload_large_file
    upload_source, file_name, file_info, part_ranges
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 480, in _find_unfinished_file
    sha1_sum = hex_sha1_of_stream(f, part_length)
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 123, in hex_sha1_of_stream
    data = input_stream.read(to_read)
MemoryError
b2_upload(/bkp/2017-08-17/system/dirs/_var_lib_mysql_.tar.gz, 2017-08-17/system/dirs/_var_lib_mysql_.tar.gz, 1502953485379): MemoryError() 
upload 2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_as30o         
upload 2017-08-17/accounts/ecotaxi/ips/related_ips                       
upload 2017-08-17/accounts/ecotaxi/mysql-timestamps/ecotaxi_aAd23        
upload 2017-08-17/accounts/ecotaxi/meta/hostname                         
upload 2017-08-17/accounts/ecotaxi/logs/radioecotaxi.com-bytes_log       
upload 2017-08-17/accounts/ecotaxi/logs/ftp.radioecotaxi.com-ftp_log      
upload 2017-08-17/system/dirs/_var_cpanel.tar.gz                           
upload 2017-08-17/accounts/ecotaxi.tar.gz                                  
ERROR:b2.console_tool:ConsoleTool command error                            
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/console_tool.py", line 1009, in run_command
    return command.run(args)
  File "build/bdist.linux-x86_64/egg/b2/console_tool.py", line 798, in run
    dry_run=args.dryRun,
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/sync/sync.py", line 262, in sync_folders
    raise CommandError('sync is incomplete')
CommandError: sync is incomplete
ERROR: sync is incomplete
svonohr commented 7 years ago

How many threads were you trying to use? It looks like python was unable to create new threads at some point. How much exactly is "a lot of memory"?

shishanyu commented 7 years ago

It seems it opened 5 threads, but IDK. I just ran b2 sync /bkp/ b2://RadioB/

with no other setting added.

svonohr commented 7 years ago

Sync defaults to 10 threads, so that should be fine. I thought maybe you were trying to run with thousands of threads. I can't really think of many reasons why creating only 10 threads would fail. Are you possibly running a machine with very limited memory? How many files does this folder and your bucket contain? According to issue #347 a sync with 1 million files in the bucket may require over 1GB of memory.

shishanyu commented 7 years ago

I think the command is decompressing the files (which do include a lot of files):

Using https://api.backblazeb2.com
Traceback (most recent call last):files   0 B   0 B/s              
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 92, in updatecache
    data = get_source(name)
error: Error -4 while decompressing data
Logged from file bucket.py, line 548
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1007-06.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001007_t0015
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1007-06.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001007_t0015 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1048-07.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001048_t0000
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1048-07.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001048_t0000 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-07.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001034_t0035
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-07.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001034_t0035 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-17.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0020
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-17.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0020 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1027-01.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001027_t0008
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1027-01.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001027_t0008 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1027-10.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001027_t0033
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1027-10.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001027_t0033 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: api000.backblazeb2.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: api000.backblazeb2.com
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-05.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0035
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-05.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0035 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-00.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0020
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-00.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0020 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-15.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0024
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-15.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0024 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1050-10.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001050_t0043
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1050-10.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001050_t0043 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-11.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001035_t0006
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-11.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001035_t0006 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: api000.backblazeb2.com
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-02.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001034_t0009
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-02.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001034_t0009 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-00.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001035_t0058
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-00.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001035_t0058 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 477, in format
    s = s + record.exc_text
MemoryError
Logged from file bucket.py, line 548
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 453, in _upload_large_file
    part_sha1_array = [interruptible_get_result(f)['contentSha1'] for f in part_futures]
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 60, in interruptible_get_result
    return future.result(timeout=1.0)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 522, in _upload_part
    upload_url, upload_auth_token = self._get_upload_part_data(file_id)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 580, in _get_upload_part_data
    response = self.api.session.get_upload_part_url(file_id)
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 322, in get_upload_part_url
    api_url, 'b2_get_upload_part_url', account_auth_token, fileId=file_id
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 96, in _translate_errors
    raise UnknownError(text)
UnknownError: Unknown error: MemoryError()
b2_upload(/bkp/2017-08-17/accounts/bemba.tar.gz, 2017-08-17/accounts/bemba.tar.gz, 1502989306492): UnknownError('MemoryError()',) Unknown error: MemoryError()
Traceback (most recent call last):/26 files   0.0 / 21.9 GB   0 B/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1014-14.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001014_t0021
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1014-14.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001014_t0021 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-17.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0014
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-17.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0014 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-17.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0005
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-17.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0005 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1035-16.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0010
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1035-16.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f200d556c747b21b3_d20170817_m203834_c000_v0001035_t0017/0010 (Caused by SSLError(SSLError(1, u'[SSL] malloc failure (_ssl.c:579)'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 453, in _upload_large_file
    part_sha1_array = [interruptible_get_result(f)['contentSha1'] for f in part_futures]
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 60, in interruptible_get_result
    return future.result(timeout=1.0)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 522, in _upload_part
    upload_url, upload_auth_token = self._get_upload_part_data(file_id)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 580, in _get_upload_part_data
    response = self.api.session.get_upload_part_url(file_id)
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 322, in get_upload_part_url
    api_url, 'b2_get_upload_part_url', account_auth_token, fileId=file_id
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 96, in _translate_errors
    raise UnknownError(text)
UnknownError: Unknown error: MemoryError()
b2_upload(/bkp/2017-08-17/accounts/estacion.tar.gz, 2017-08-17/accounts/estacion.tar.gz, 1502974507292): UnknownError('MemoryError()',) Unknown error: MemoryError()
ERROR:b2.sync.action:an exception occurred in a sync action   2.26 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 429, in _upload_large_file
    upload_source, file_name, file_info, part_ranges
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 466, in _find_unfinished_file
    for file_ in self.list_unfinished_large_files():
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 273, in list_unfinished_large_files
    self.id_, start_file_id, batch_size
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 383, in list_unfinished_large_files
    maxFileCount=max_file_count
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 279, in post_content_return_json
    return json.loads(response.content.decode('utf-8'))
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
MemoryError
b2_upload(/bkp/2017-08-17/accounts/lacajama.tar.gz, 2017-08-17/accounts/lacajama.tar.gz, 1502995454664): MemoryError() 
Traceback (most recent call last):/26 files   0.0 / 21.9 GB   6.76 kB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):/26 files   0.0 / 21.9 GB   26.2 kB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1014-01.backblaze.com/b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001014_t0037
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 394, in _upload_small_file
    HEX_DIGITS_AT_END, file_info, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 458, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1014-01.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_file/92f33e3b4482eaf25ad80f15/c000_v0001014_t0037 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 429, in _upload_large_file
    upload_source, file_name, file_info, part_ranges
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 466, in _find_unfinished_file
    for file_ in self.list_unfinished_large_files():
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 273, in list_unfinished_large_files
    self.id_, start_file_id, batch_size
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 383, in list_unfinished_large_files
    maxFileCount=max_file_count
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 96, in _translate_errors
    raise UnknownError(text)
UnknownError: Unknown error: MemoryError('out of memory',)
b2_upload(/bkp/2017-08-17/accounts/lamatera.tar.gz, 2017-08-17/accounts/lamatera.tar.gz, 1502977029274): UnknownError("MemoryError('out of memory',)",) Unknown error: MemoryError('out of memory',)
ERROR:b2.sync.action:an exception occurred in a sync action   130 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 453, in _upload_large_file
    part_sha1_array = [interruptible_get_result(f)['contentSha1'] for f in part_futures]
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 60, in interruptible_get_result
    return future.result(timeout=1.0)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 522, in _upload_part
    upload_url, upload_auth_token = self._get_upload_part_data(file_id)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 580, in _get_upload_part_data
    response = self.api.session.get_upload_part_url(file_id)
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 322, in get_upload_part_url
    api_url, 'b2_get_upload_part_url', account_auth_token, fileId=file_id
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 96, in _translate_errors
    raise UnknownError(text)
UnknownError: Unknown error: MemoryError()
b2_upload(/bkp/2017-08-17/accounts/aguiarm.tar.gz, 2017-08-17/accounts/aguiarm.tar.gz, 1502973714595): UnknownError('MemoryError()',) Unknown error: MemoryError()
ERROR:b2.sync.action:an exception occurred in a sync action   129 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 453, in _upload_large_file
    part_sha1_array = [interruptible_get_result(f)['contentSha1'] for f in part_futures]
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 60, in interruptible_get_result
    return future.result(timeout=1.0)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 522, in _upload_part
    upload_url, upload_auth_token = self._get_upload_part_data(file_id)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 580, in _get_upload_part_data
    response = self.api.session.get_upload_part_url(file_id)
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 322, in get_upload_part_url
    api_url, 'b2_get_upload_part_url', account_auth_token, fileId=file_id
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='api000.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v1/b2_get_upload_part_url (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
b2_upload(/bkp/2017-08-17/accounts/acita.tar.gz, 2017-08-17/accounts/acita.tar.gz, 1502986592415): B2ConnectionError("HTTPSConnectionPool(host='api000.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v1/b2_get_upload_part_url (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))",) Connection error: HTTPSConnectionPool(host='api000.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v1/b2_get_upload_part_url (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.b2http:_translate_errors has intercepted an unexpected exception
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 47, in _translate_errors
    response = fcn()
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 268, in do_post
    response = self.session.post(url, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 494, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 424, in prepare_request
    auth = get_netrc_auth(request.url)
  File "/usr/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/utils.py", line 164, in get_netrc_auth
    from netrc import netrc, NetrcParseError
MemoryError
shishanyu commented 7 years ago

The system has 16GB of memory and I don't think it's memory constrained. I eliminated a folder that consisted of a backup file decompressed and it worked kind of fine, except it ignored certain files to which it didn't have access to. I added access to my account with setfacl and it gained access but then it gave me the errors I just posted.

bwbeach commented 7 years ago

The error decompressing data came while trying to log an exception. I think it was trying to get the source code for the line where the exception happened:

  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 92, in updatecache
    data = get_source(name)
error: Error -4 while decompressing data
Logged from file bucket.py, line 548

Did you install b2 using pip install b2?

shishanyu commented 7 years ago

Yes. I later tried installing it with this git repository, but didn't uninstall the pip install of b2.

bwbeach commented 7 years ago

Would you be willing to try syncing with just one thread and see what happens? The option is: --threads 1

ppolewicz commented 7 years ago

@shishanyu there is clearly an issue with memory allocation on that system. What does it say when you run free -m? Do you have some memory quota set (by ulimit or something)?

Your memory can also be fragmented, which means there is a lot of it free, but there are not enough consecutive free bytes, so when we are trying to allocate memory for something, the kernel looks through the whole memory for a chunk big enough for this and finds nothing. This is unlikely, but still possible.

B2 CLI is not decompressing your data. It doesn't have such capability. The stacktrace that you have seen, including a message about this, is actually about something else (such as decompressing of a message from server - but not something that was on disk).

shishanyu commented 7 years ago

@bwbeach I ran it with the --threads 1 option, and it's working steadily. I'll tell you once I encounter an error or it finishes.

@ppolewicz 

               total        used        free      shared  buff/cache   available
Mem:          16046        2109         415         401       13521       13065
Swap:          7628         682        6946
shishanyu commented 7 years ago

It seems it worked fine with --threads 1, everything finished alright. Maybe you could consider adding a warning when these errors happen, kind of: "try running with less threads by adding --threads 1" to your command.

ppolewicz commented 7 years ago

We'd rather fix the bug :)

Would it be possible for you to run it again with multiple threads and check how much memory is really consumed by B2 CLI? We'd like to confirm that this is really a memory utilization issue.

shishanyu commented 7 years ago

I ran it without the --threads 1 and right now it's working BUT, yesterday I moved out an account that was consuming a lot of CPU and resources so the server is much more relaxed. I don't think I can replicate the bug now, because yesterday the server was at a load average of around 30-35 (Atom N2800 (2 cores)) and now it's around 1.5 to 2 with b2 running with the default number of threads.

Right now it's running fine and smoothly.

shishanyu commented 7 years ago

current free -m with b2 running with default threads and load average of about 1.6:

              total        used        free      shared  buff/cache   available
Mem:          16046        1932         465          86       13648       13558
Swap:          7628         884        6744
shishanyu commented 7 years ago

Nevermind, I'm still getting the error:

Using https://api.backblazeb2.com
ERROR:b2.sync.action:an exception occurred in a sync action   194 kB/s 
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 447, in _upload_large_file
    ) for (part_index, part_range) in enumerate(part_ranges)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/crex.tar.gz, 2017-08-17/accounts/crex.tar.gz, 1503074056560): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   471 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 447, in _upload_large_file
    ) for (part_index, part_range) in enumerate(part_ranges)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/fsmex.tar.gz, 2017-08-17/accounts/fsmex.tar.gz, 1503073747724): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   471 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 447, in _upload_large_file
    ) for (part_index, part_range) in enumerate(part_ranges)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/grupomb.tar.gz, 2017-08-17/accounts/grupomb.tar.gz, 1503066528679): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   466 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/josehernandez.tar.gz, 2017-08-17/accounts/josehernandez.tar.gz, 1503069690380): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   616 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/klean.tar.gz, 2017-08-17/accounts/klean.tar.gz, 1503060427209): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   615 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/mxoot.tar.gz, 2017-08-17/accounts/mxoot.tar.gz, 1503059983003): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   615 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/mst.tar.gz, 2017-08-17/accounts/mst.tar.gz, 1503067156849): error("can't start new thread",) can't start new thread
ERROR:b2.sync.action:an exception occurred in a sync action   613 kB/s
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 368, in upload
    progress_listener
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/tutecho.tar.gz, 2017-08-17/accounts/tutecho.tar.gz, 1503060056298): error("can't start new thread",) can't start new thread
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-04.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0026
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-04.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0026 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-19.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0019
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-19.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0019 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-15.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0030
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-15.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0030 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-10.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0036
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-10.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0036 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 429, in _upload_large_file
    upload_source, file_name, file_info, part_ranges
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 480, in _find_unfinished_file
    sha1_sum = hex_sha1_of_stream(f, part_length)
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 123, in hex_sha1_of_stream
    data = input_stream.read(to_read)
MemoryError
b2_upload(/bkp/2017-08-17/accounts/copyserv.tar.gz, 2017-08-17/accounts/copyserv.tar.gz, 1503059952469): MemoryError() 
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 447, in _upload_large_file
    ) for (part_index, part_range) in enumerate(part_ranges)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python2.7/threading.py", line 746, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
b2_upload(/bkp/2017-08-17/accounts/zeigen.tar.gz, 2017-08-17/accounts/zeigen.tar.gz, 1503069871873): error("can't start new thread",) can't start new thread
Traceback (most recent call last):/18 files   0.0 / 44.8 GB   3.21 MB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Unhandled exception in thread started by 
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-09.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0000
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-09.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0000 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 824, in __bootstrap_inner
    (self.name, _format_exc()))
MemoryError
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-18.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0026
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-18.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0026 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-09.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0021
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-09.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0021 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-19.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0035
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-19.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0035 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):/18 files   0.0 / 44.8 GB   7.85 MB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):/18 files   0.0 / 44.8 GB   8.08 MB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-12.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0021
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-12.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0021 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-13.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0022
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-13.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0022 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
Unhandled exception in thread started by 
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
  File "/usr/lib64/python2.7/threading.py", line 824, in __bootstrap_inner
MemoryError
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 453, in _upload_large_file
    part_sha1_array = [interruptible_get_result(f)['contentSha1'] for f in part_futures]
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 60, in interruptible_get_result
    return future.result(timeout=1.0)
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 522, in _upload_part
    upload_url, upload_auth_token = self._get_upload_part_data(file_id)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 580, in _get_upload_part_data
    response = self.api.session.get_upload_part_url(file_id)
  File "build/bdist.linux-x86_64/egg/b2/session.py", line 38, in wrapper
    return f(api_url, account_auth_token, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 322, in get_upload_part_url
    api_url, 'b2_get_upload_part_url', account_auth_token, fileId=file_id
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 138, in _post_json
    return self.b2_http.post_json_return_json(url, headers, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 299, in post_json_return_json
    return self.post_content_return_json(url, headers, data, try_count, params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 95, in _translate_errors
    logger.exception('_translate_errors has intercepted an unexpected exception')
  File "/usr/lib64/python2.7/logging/__init__.py", line 1182, in exception
    self.error(msg, *args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1175, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 879, in emit
    self.handleError(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 802, in handleError
    None, sys.stderr)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
b2_upload(/bkp/2017-08-17/accounts/sergio.tar.gz, 2017-08-17/accounts/sergio.tar.gz, 1503074467424): MemoryError() 
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-18.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0023
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-18.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0023 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
Traceback (most recent call last):/18 files   0.0 / 44.8 GB   7.97 MB/s
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 472, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib64/python2.7/logging/__init__.py", line 438, in formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib64/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
    lines = fp.readlines()
MemoryError
Logged from file b2http.py, line 95
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1010-16.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f2011c8e2b5ec17d1_d20170818_m203308_c000_v0001010_t0039/0011
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 96, in _translate_errors
    raise UnknownError(text)
UnknownError: Unknown error: MemoryError()
ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1034-05.backblaze.com/b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0005
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 539, in _upload_part
    HEX_DIGITS_AT_END, hashing_stream
  File "build/bdist.linux-x86_64/egg/b2/raw_api.py", line 470, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 272, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 116, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "build/bdist.linux-x86_64/egg/b2/b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
B2ConnectionError: Connection error: HTTPSConnectionPool(host='pod-000-1034-05.backblaze.com', port=443): Max retries exceeded with url: /b2api/v1/b2_upload_part/4_z92f33e3b4482eaf25ad80f15_f20025e89063c9058_d20170818_m203307_c000_v0001034_t0025/0005 (Caused by SSLError(IOError(12, 'Cannot allocate memory'),))
ERROR:b2.sync.action:an exception occurred in a sync action
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 42, in run
    self.do_action(bucket, reporter)
  File "build/bdist.linux-x86_64/egg/b2/sync/action.py", line 84, in do_action
    progress_listener=SyncFileReporter(reporter)
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 373, in upload
    upload_source, file_name, content_type, file_info, progress_listener
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 429, in _upload_large_file
    upload_source, file_name, file_info, part_ranges
  File "build/bdist.linux-x86_64/egg/b2/bucket.py", line 480, in _find_unfinished_file
    sha1_sum = hex_sha1_of_stream(f, part_length)
  File "build/bdist.linux-x86_64/egg/b2/utils.py", line 123, in hex_sha1_of_stream
    data = input_stream.read(to_read)
MemoryError
b2_upload(/bkp/2017-08-17/accounts/fpm.tar.gz, 2017-08-17/accounts/fpm.tar.gz, 1503069536363): MemoryError() 
upload 2017-08-17/accounts/hilos.tar.gz                                
ERROR:b2.console_tool:ConsoleTool command interrupt                      
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/b2/console_tool.py", line 1009, in run_command
    return command.run(args)
  File "build/bdist.linux-x86_64/egg/b2/console_tool.py", line 798, in run
    dry_run=args.dryRun,
  File "/usr/lib/python2.7/site-packages/logfury-0.1.2-py2.7.egg/logfury/v0_1/trace_call.py", line 84, in wrapper
    return function(*wrapee_args, **wrapee_kwargs)
  File "build/bdist.linux-x86_64/egg/b2/sync/sync.py", line 260, in sync_folders
    sync_executor.shutdown()
  File "build/bdist.linux-x86_64/egg/b2/sync/sync.py", line 196, in shutdown
    self.executor.shutdown()
  File "/usr/lib/python2.7/site-packages/futures-3.1.1-py2.7.egg/concurrent/futures/thread.py", line 148, in shutdown
    t.join(sys.maxint)
  File "/usr/lib64/python2.7/threading.py", line 961, in join
    self.__block.wait(delay, balancing)
  File "/usr/lib64/python2.7/threading.py", line 361, in wait
    _sleep(delay)
  File "build/bdist.linux-x86_64/egg/b2/console_tool.py", line 68, in keyboard_interrupt_handler
    raise KeyboardInterrupt()
KeyboardInterrupt

Interrupted.  Shutting down...
shishanyu commented 7 years ago

And the server was at 0.5 load average.

ppolewicz commented 7 years ago

Could you please paste the result of ulimit -a from the account from which you have ran B2 CLI?

shishanyu commented 7 years ago 
core file size          (blocks, -c) 200000
data seg size           (kbytes, -d) 200000
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64105
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 200000
open files                      (-n) 100
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 35
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
ppolewicz commented 7 years ago

It looks like you have a memory limit set to 200MB, which explains MemoryError . You have a process limit set to 35, which explains error: can't start new thread. Environment is mis-configured.