IndexError: list index out of range

[Wimi2]

Describe the bug YAPO e+ stops adding files after some (a lot) of files.

EDIT: It continues adding files after restarting YAPO and re-adding the folder.

Scanning...   --> A result was returned using method 1. Parsing JSON...
Internal Server Error: /add-items/
Traceback (most recent call last):
  File "django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "django\core\handlers\base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django\core\handlers\base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django\views\decorators\csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "django\views\generic\base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework\views.py", line 505, in dispatch
  File "rest_framework\views.py", line 465, in handle_exception
  File "rest_framework\views.py", line 476, in raise_uncaught_exception
  File "rest_framework\views.py", line 502, in dispatch
  File "views.py", line 1329, in get
  File "addScenes.py", line 64, in get_files
  File "addScenes.py", line 262, in create_scene
  File "scrapers\scanners.py", line 148, in tpdb
IndexError: list index out of range
"GET /add-items/?actorsToAdd=&createSampleVideo=false&folderToAddPath=D:%5C%5C&sceneTagsToAdd=&websitesToAdd= HTTP/1.1" 500 15555

[cooperdk]

When you say a lot of files, do you mean more than 13000? YAPO is not tested past this amount, but has been succesful within 13000 scans in one go.

In some cases, though, it is stopped due to outside factors (network access, etc.

This is probably one of those cases, because there should be no trouble with the code if it continues after the folder is re-added.

I will take a look at the error though, and add an error handler to the code in question.

If you know Python, you can wrap the conditional statements around line 148 in videos/scrapers/scanners.py around an error catcher.

They must follow indentation and push the if statements four spaces further out.

try:
    ...
except:
    pass

And they should be just after the print and continue thru this code:

        if found == 1 or found == 2:

            print(f"  --> A result was returned using method {found}. Parsing JSON...")
            site_logo = ""
            if "title" in response['data'][0].keys():
                title = response['data'][0]['title']
                print(f"  --> Title is {title}")
            if "description" in response['data'][0].keys():
                description = response['data'][0]['description']
            if "site" in response['data'][0].keys():
                if "name" in response['data'][0]['site'].keys():
                    site = response['data'][0]['site']['name']
                    site_logo = response['data'][0]['site']['logo']
                if "tags" in response['data'][0].keys():
                    scenetags = response['data'][0]['tags']
            if "date" in response['data'][0].keys():
                release_date = response['data'][0]['date']
            if "id" in response['data'][0].keys():
                tpdb_id = response['data'][0]['id']
            if description is not None: current_scene.description = description
            if release_date is not None:
                #print(release_date)
                try:
                    release_date=datetime.strptime(release_date, "%Y-%m-%d").date()
                #release_date = strftime(release_date) + " 00:00:00"
                #release_date = datetime.strptime(release_date, "%Y-%m-%d %H:%M:%S")
                    print(release_date)
                except:
                    release_date = None
                if release_date is not None:
                    current_scene.release_date = release_date

[Wimi2]

I cannot confirm the files amount atm, because I don't have access to my computer, but it is very likely, that there are more than 13k files (with a lot of smaller files, which are distorting the numbers). The files are on an internal HDD (not network etc.), so I don't think, that the error happened due to connectivity issues.

I also don't think, that there are problems with the code in general, just some variable running out of index_space or something. You can make this one a low priority, I think, because the user can just restart YAPO and re-add the HDD/folder again with YAPO skipping already existing files, but it's good to know, that YAPO has got a upper adding limit (in one go).

[cooperdk]

It shouldn't have, there might also be an error in one of your files (I have seen this error in these cases). I will have to check up on the code again, but there may be many causes to this issue.

[Wimi2]

I re-added my files some one or two days ago and there I encountered this bug twice in the process. YAPO says, it has around 7k files, so it wasn't the sheer numbers of files that have caused the bug.

I couldn't find any interesting errors in the log file either, so yeah, it will be hard to find out the faulty file in particular. One solution would be adding everything again and looking at what file exactly YAPO does stop, but I would rather not do it right now. It's very time consuming.

All, that I can think of other than your faulty file is a file path that is too long.

[cooperdk]

It could be. I have experienced a few odd command traces from Python where there was no clear error to debug from. This is in part due to the fact that YAPO wasn't originally developed to carry over errors from exceptions. If it continues after an error (like you mention, a path that's too long), the resulting variable will be empty and then is no data to handle, which causes an error if the data handler is not handled by an exception.

One of the caveats of the original YAPO is inconsistent coding and poor error handling. I am in the process of introducing a logger function, and it's added to functions/methods whenever I have a file open anyway. This logger (which outputs errors to data/yapo-eplus.log) will sometimes point out the cause of an error.

[cooperdk]

Closed, expected to not really be an issue anymore.