search

boto / s3transfer

Amazon S3 Transfer Manager for Python
Apache License 2.0
209 stars 133 forks source link

TestNonMultipartUpload is probably flaky on x86 #116

Closed andrey-utkin closed 3 years ago

andrey-utkin commented 5 years ago

Hi, It has occured two times in last few months that s3transfer fails a particular test, TestNonMultipartUpload, on 32-bit x86 machine. The machine has 3 GB RAM, 1 GB was free at the time of the test. I guess it could be a legitimate resource shortage. Also I guess you don't have much interest in 32-bit x86 and in testing on it in particular.

May I ask you the developers to review this test, confirming that the test has reasons to be such a memory hog. Once confirmed, we can just exclude this one from being run on 32 bits. Maybe you could even add such an exclusion into your test suite.

Thanks in advance.


======================================================================
ERROR: test_uses_provided_osutil (tests.functional.test_upload.TestNonMultipartUpload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13/tests/functional/test_upload.py", line 231,
 in test_uses_provided_osutil
    future = self.manager.upload(self.filename, self.bucket, self.key)
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/manager.py", line
308, in upload
    call_args, UploadSubmissionTask, extra_main_kwargs)
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/manager.py", line
464, in _submit_transfer
    main_kwargs=main_kwargs
  File "/var/tmp/portage/dev-python/s3transfer-0.1.13/work/s3transfer-0.1.13-python2_7/lib/s3transfer/futures.py", line
435, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 122, in submit
    self._adjust_thread_count()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 138, in _adjust_thread_count
    t.start()
  File "/usr/lib/python2.7/threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread

See https://bugs.gentoo.org/660474 , the original report. Then, it happened to pass on a couple of other occasions, but then it reoccured, see here: https://paste.pound-python.org/show/cJwK8BMbkAHbFfLInDeq/ .

joguSD commented 5 years ago

I think this is a legitimate resource shortage, but nothing in that particular test sticks out to me as being particularly resource heavy. I wonder if the test suite is having trouble cleaning up threads. I imagine running this one test alone likely wouldn't fail.