search

BD2KGenomics / s3am

A fast, parallel, streaming multipart uploader for S3
Other
13 stars 6 forks source link

Failed assertion if file size is exactly 10 MiB (10 * 1024 * 1024) #17

Closed hannes-ucsc closed 9 years ago

hannes-ucsc commented 9 years ago 
mesosbox@ip-172-31-19-218:~$ dd if=/dev/urandom of=foo bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.950645 s, 11.0 MB/s
mesosbox@ip-172-31-19-218:~$ s3am upload --verbose file:///home/mesosbox/foo hannes-foo foo 
2015-09-28 23:00:05,831 operations(9545) Streaming file:///home/mesosbox/foo
2015-09-28 23:00:05,832 operations(9545) part 0: dispatching
2015-09-28 23:00:05,832 operations(9549) part 0: downloading
2015-09-28 23:00:05,838 operations(9545) part 1: dispatching
2015-09-28 23:00:05,838 operations(9548) part 1: downloading
2015-09-28 23:00:05,843 operations(9545) part 2: dispatching
2015-09-28 23:00:05,843 operations(9549) part 0: downloaded 5.0 MB (5242880 bytes)
2015-09-28 23:00:05,844 operations(9549) part 0: uploading
2015-09-28 23:00:05,846 operations(9548) part 1: downloaded 5.0 MB (5242880 bytes)
2015-09-28 23:00:05,846 operations(9548) part 1: uploading
2015-09-28 23:00:06,113 operations(9548) part 1: uploaded 5.0 MB (5242880 bytes)
2015-09-28 23:00:06,115 operations(9548) part 2: downloading
2015-09-28 23:00:06,115 operations(9545) part 3: dispatching
2015-09-28 23:00:06,116 operations(9548) part 2: downloaded 0.0 B (0 bytes)
2015-09-28 23:00:06,116 operations(9548) part 3: downloading
2015-09-28 23:00:06,116 operations(9545) part 4: dispatching
2015-09-28 23:00:06,117 operations(9548) part 3: downloaded 0.0 B (0 bytes)
2015-09-28 23:00:06,117 operations(9548) part 4: downloading
2015-09-28 23:00:06,117 operations(9548) part 4: downloaded 0.0 B (0 bytes)
2015-09-28 23:00:06,164 operations(9549) part 0: uploaded 5.0 MB (5242880 bytes)
Traceback (most recent call last):
  File "/usr/local/bin/s3am", line 9, in <module>
    load_entry_point('s3am==1.0a1.dev14', 'console_scripts', 's3am')()
  File "/usr/local/lib/python2.7/dist-packages/s3am/cli.py", line 31, in try_main
    main( args )
  File "/usr/local/lib/python2.7/dist-packages/s3am/cli.py", line 60, in main
    operation.run(  )
  File "/usr/local/lib/python2.7/dist-packages/s3am/operations.py", line 232, in run
    self._sanity_check( completed_parts )
  File "/usr/local/lib/python2.7/dist-packages/s3am/operations.py", line 494, in _sanity_check
    assert s( *groups[ 0 ] ) or l( *groups[ 0 ] )
AssertionError
mesosbox@ip-172-31-19-218:~$ echo 1 >> foo
mesosbox@ip-172-31-19-218:~$ s3am upload --verbose file:///home/mesosbox/foo hannes-foo foo 
error: Transfer failed. There is a pending upload. If you would like to resume that upload, run s3am again with --resume. If you would like to cancel the upload, use 's3am cancel hannes-foo foo'. Note that pending uploads incur storage fees.
mesosbox@ip-172-31-19-218:~$ s3am cancel --prefix hannes-foo ""
mesosbox@ip-172-31-19-218:~$ s3am upload --verbose file:///home/mesosbox/foo hannes-foo foo 
2015-09-28 23:01:09,512 operations(9559) Streaming file:///home/mesosbox/foo
2015-09-28 23:01:09,513 operations(9559) part 0: dispatching
2015-09-28 23:01:09,513 operations(9563) part 0: downloading
2015-09-28 23:01:09,519 operations(9559) part 1: dispatching
2015-09-28 23:01:09,519 operations(9562) part 1: downloading
2015-09-28 23:01:09,524 operations(9559) part 2: dispatching
2015-09-28 23:01:09,524 operations(9562) part 1: downloaded 5.0 MB (5242880 bytes)
2015-09-28 23:01:09,525 operations(9562) part 1: uploading
2015-09-28 23:01:09,527 operations(9563) part 0: downloaded 5.0 MB (5242880 bytes)
2015-09-28 23:01:09,527 operations(9563) part 0: uploading
2015-09-28 23:01:09,914 operations(9563) part 0: uploaded 5.0 MB (5242880 bytes)
2015-09-28 23:01:09,916 operations(9563) part 2: downloading
2015-09-28 23:01:09,916 operations(9559) part 3: dispatching
2015-09-28 23:01:09,917 operations(9563) part 2: downloaded 2.0 B (2 bytes)
2015-09-28 23:01:09,917 operations(9563) part 2: uploading
2015-09-28 23:01:09,921 operations(9562) part 1: uploaded 5.0 MB (5242880 bytes)
2015-09-28 23:01:09,922 operations(9562) part 3: downloading
2015-09-28 23:01:09,922 operations(9559) part 4: dispatching
2015-09-28 23:01:09,923 operations(9562) part 3: downloaded 0.0 B (0 bytes)
2015-09-28 23:01:09,923 operations(9562) part 4: downloading
2015-09-28 23:01:09,923 operations(9562) part 4: downloaded 0.0 B (0 bytes)
2015-09-28 23:01:09,977 operations(9563) part 2: uploaded 2.0 B (2 bytes)
2015-09-28 23:01:10,091 operations(9559) Completed file:///home/mesosbox/foo