10-time retry could not be enough for veeery large uploads

[alpianon]

https://github.com/fossology/fossology-python/blob/fa8ed27515782cb4caba16514d57a219b503ee89/fossology/uploads.py#L20

10-time retry to check if upload has been unpacked means 10 minutes (more or less). However, in case on very large packages, this may be not enough...

[alpianon]

probably it would be better to check job status instead?

[deveaud-m]

probably it would be better to check job status instead?

That's the purpose of the Retry-After headers sent by GET requests to the endpoint /uploads/{id}, if the unpack/adj2nest jobs are not finished, the server returns a 503. This is no tdocumented in the API specification yet, this needs some upstream fix in https://github.com/fossology/fossology, I'll take care of this.

I've propose a solution for this issue in #43.

[deveaud-m]

@alpianon please test the newest version (e.g. from https://pypi.org/project/fossology/0.1.4/) and report back if your problem could be solved.

[leimaohui]

I installed 0.1.4, and met the following question:

     0164:    try:
 *** 0165:        upload = foss.upload_file(
     0166:        folder,
     0167:        file=filepath,
     0168:        access_level=AccessLevel.PUBLIC,
     0169:        )
File: '/ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages/fossology/uploads.py', lineno: 188, function: upload_file
     0184:            return
     0185:
     0186:        if response.status_code == 201:
     0187:            try:
 *** 0188:                upload = self.detail_upload(response.json()["message"], wait_time)
     0189:                logger.info(
     0190:                    f"Upload {upload.uploadname} ({upload.filesize}) "
     0191:                    f"has been uploaded on {upload.uploaddate}"
     0192:                )
File: '/ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages/tenacity/__init__.py', lineno: 329, function: wrapped_f
     0325:        :param f: A function to wraps for retrying.
     0326:        """
     0327:        @_utils.wraps(f)
     0328:        def wrapped_f(*args, **kw):
 *** 0329:            return self.call(f, *args, **kw)
     0330:
     0331:        def retry_with(*args, **kwargs):
     0332:            return self.copy(*args, **kwargs).wraps(f)
     0333:
File: '/ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages/tenacity/__init__.py', lineno: 409, function: call
     0405:
     0406:        retry_state = RetryCallState(
     0407:            retry_object=self, fn=fn, args=args, kwargs=kwargs)
     0408:        while True:
 *** 0409:            do = self.iter(retry_state=retry_state)
     0410:            if isinstance(do, DoAttempt):
     0411:                try:
     0412:                    result = fn(*args, **kwargs)
     0413:                except BaseException:
File: '/ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages/tenacity/__init__.py', lineno: 369, function: iter
     0365:                return self.retry_error_callback(retry_state=retry_state)
     0366:            retry_exc = self.retry_error_cls(fut)
     0367:            if self.reraise:
     0368:                raise retry_exc.reraise()
 *** 0369:            six.raise_from(retry_exc, fut.exception())
     0370:
     0371:        if self.wait:
     0372:            sleep = self.wait(retry_state=retry_state)
     0373:        else:
File: '<string>', lineno: 3, function: raise_from
  File "<string>", line 3, in raise_from

Exception: tenacity.RetryError: RetryError[<Future at 0x7f1f424e0e50 state=finished raised TryAgain>]

$ pip3 install fossology
Requirement already satisfied: fossology in /ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages (0.1.4)
Requirement already satisfied: tenacity>=6.0.0 in /ubinux-dev/common/home/ubinux/.local/lib/python3.8/site-packages (from fossology) (6.2.0)
Requirement already satisfied: requests>=2.22.0 in /usr/lib/python3/dist-packages (from fossology) (2.22.0)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from tenacity>=6.0.0->fossology) (1.14.0)
$

Is this the same question?

[deveaud-m]

@leimaohui you can now use the option wait_time when calling upload_file(), see the docs for details: https://fossology.github.io/fossology-python/uploads.html#fossology.uploads.Uploads.upload_file.

It allows you to define the time interval to wait until the next retry is performed, by default Retry-After is 60 seconds, if you set wait_time=120 the REST call by be performed every 2 minutes (120 seconds), max 10 times.

[leimaohui]

@deveaud-m Thank you. It works.

[deveaud-m]

@leimaohui thanks for reporting, closing this issue!