search

vgrem / Office365-REST-Python-Client

Microsoft 365 & Microsoft Graph Library for Python
MIT License
1.34k stars 335 forks source link

Error: "List index out of range" when searching/downloading a file #912

Open GrumeiA opened 1 week ago

GrumeiA commented 1 week ago

Intro:

Hello. First of all, thank you for the effort that you put into this library. Is is very usefull for me and many other users.

Current situation:

Since a few weeks ago I am encountering a failure which leads to this library. I am trying to search or download automatically some files from SharePoint (Excel files, if it matters). My code has a retry in case of sharepoint failure, meaning that it will redo the python object, with a fresh connection and then retry to download again. This procedure is redone 3 times. If download still fails, then it will abandon the download process.

Problem:

Sometimes the search/download fails, even after 3 retry attempts. The behaviour is seen random, with no specific conditions. If I give it a retry a few hours later, everything works fine.

Please have a look over the trace that I get from Office365 library:

[07/11 11:22:20][D]|Default mandatory relative path: VEDVerification| (py_sharepoint.py:276)
[07/11 11:22:20][D]|Custom mandatory path: SRSx_TestData/SRSx-GEN/TDB| (py_sharepoint.py:277)
[07/11 11:22:20][D](utils.py:24)|list index out of range
Traceback (most recent call last):
  File "utils\utils.py", line 22, in safe_execute_sharepoint_operation
  File "py_sharepoint\py_sharepoint.py", line 283, in search_file
  File "office365\runtime\client_result.py", line 53, in execute_query
  File "office365\runtime\client_runtime_context.py", line 173, in execute_query
  File "office365\runtime\client_request.py", line 37, in execute_query
  File "office365\runtime\client_request.py", line 47, in execute_request_direct
  File "office365\runtime\types\event_handler.py", line 41, in notify
  File "office365\sharepoint\client_context.py", line 281, in _authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 239, in authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 196, in _authenticate
  File "office365\runtime\auth\providers\saml_token_provider.py", line 85, in authenticate_request
  File "office365\runtime\auth\providers\saml_token_provider.py", line 97, in ensure_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 109, in get_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 160, in _acquire_service_token_from_adfs
IndexError: list index out of range
[07/11 11:22:20][D]|Connection to https://continental.sharepoint.com/teams/team_10039407 was reinitialized.| (py_sharepoint.py:119)
[07/11 11:22:20][D]|Default mandatory relative path: VEDVerification| (py_sharepoint.py:276)
[07/11 11:22:20][D]|Custom mandatory path: SRSx_TestData/SRSx-GEN/TDB| (py_sharepoint.py:277)
[07/11 11:22:21][D](utils.py:24)|list index out of range
Traceback (most recent call last):
  File "utils\utils.py", line 22, in safe_execute_sharepoint_operation
  File "py_sharepoint\py_sharepoint.py", line 283, in search_file
  File "office365\runtime\client_result.py", line 53, in execute_query
  File "office365\runtime\client_runtime_context.py", line 173, in execute_query
  File "office365\runtime\client_request.py", line 37, in execute_query
  File "office365\runtime\client_request.py", line 47, in execute_request_direct
  File "office365\runtime\types\event_handler.py", line 41, in notify
  File "office365\sharepoint\client_context.py", line 281, in _authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 239, in authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 196, in _authenticate
  File "office365\runtime\auth\providers\saml_token_provider.py", line 85, in authenticate_request
  File "office365\runtime\auth\providers\saml_token_provider.py", line 97, in ensure_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 109, in get_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 160, in _acquire_service_token_from_adfs
IndexError: list index out of range
[07/11 11:22:21][D]|Connection to https://continental.sharepoint.com/teams/team_10039407 was reinitialized.| (py_sharepoint.py:119)
[07/11 11:22:21][D]|Default mandatory relative path: VEDVerification| (py_sharepoint.py:276)
[07/11 11:22:21][D]|Custom mandatory path: SRSx_TestData/SRSx-GEN/TDB| (py_sharepoint.py:277)
[07/11 11:22:21][D](utils.py:24)|list index out of range
Traceback (most recent call last):
  File "utils\utils.py", line 22, in safe_execute_sharepoint_operation
  File "py_sharepoint\py_sharepoint.py", line 283, in search_file
  File "office365\runtime\client_result.py", line 53, in execute_query
  File "office365\runtime\client_runtime_context.py", line 173, in execute_query
  File "office365\runtime\client_request.py", line 37, in execute_query
  File "office365\runtime\client_request.py", line 47, in execute_request_direct
  File "office365\runtime\types\event_handler.py", line 41, in notify
  File "office365\sharepoint\client_context.py", line 281, in _authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 239, in authenticate_request
  File "office365\runtime\auth\authentication_context.py", line 196, in _authenticate
  File "office365\runtime\auth\providers\saml_token_provider.py", line 85, in authenticate_request
  File "office365\runtime\auth\providers\saml_token_provider.py", line 97, in ensure_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 109, in get_authentication_cookie
  File "office365\runtime\auth\providers\saml_token_provider.py", line 160, in _acquire_service_token_from_adfs
IndexError: list index out of range
[07/11 11:22:21][D]|Connection to https://continental.sharepoint.com/teams/team_10039407 was reinitialized.| (py_sharepoint.py:119)
[07/11 11:22:21][E](utils.py:33)|Failed to execute method search_file for 3 times.

For debug details I can also offer the code used in this case (for search):

    def search_file(self, file_name: str, mandatory_path: Optional[str] = None) -> Optional[str]:
        """
        Method searches a file matching the given name and downloads it to local folder.
        If more than one file are found with the same file name, then only the first will be chosen.

        Parameters
        ----------
        file_name: str
                   Full name of the file to search, extension (.txt, .xlsx) included.
                   Ex: "TDB_ATRT_EBT-VOLVO-QC3_MKC1_EBT_Safes.xlsx"
        mandatory_path: str, default=""
                        Extra mandatory path within SharePoint that the found file has to be in.
                        Ex: "SRSx_TestData/SRSx-GEN/MLDB".
                        Say that there two files with same name "MLDB.xlsx" in two different folders:
                            "/Team1/MLDB.xlsx"
                            "/Team2/MLDB.xlsx"
                        If you need the file from 'Team2' folder, you can specify 'mandatory_path="Team2"'

        Returns
        -------
        None or str
            None if no file has been found, otherwise a string containing the local relative URL to the found file.

        Examples
        --------
        >>> conn = SharepointConnection("user@contiwan.com", "pass")
        >>> conn.search_file("TDB_ATRT_EBT-VOLVO-QC1_MKC1_GENERIC_milestone_Valentina.xlsx", "EBT-VOLVO-QC1")
        "VEDVerification/SRSx_TestData/EBT-VOLVO-QC1/TDB/TDB_ATRT_EBT-VOLVO-QC1_MKC1_GENERIC_milestone_Valentina.xlsx"
        """
        logger.debug(f"Default mandatory relative path: {self.mandatory_rel_path}")
        logger.debug(f"Custom mandatory path: {mandatory_path}")
        substrings_to_check = [self.mandatory_rel_path, file_name]
        if mandatory_path is not None:
            substrings_to_check.append(mandatory_path)

        # search part
        result = self.context.search.post_query(query_text=f"FileName:'{file_name}'").execute_query()
        for row in result.value.PrimaryQueryResult.RelevantResults.Table.Rows:
            file_url = str(row.Cells["Path"])
            if not any(sub_str in file_url for sub_str in substrings_to_check):
                continue
            logger.debug(f"Found '{file_url}'")
            break
        else:
            logger.error(f"No file was found with name {file_name}")
            return None

        return self.mandatory_rel_path + file_url.split(self.mandatory_rel_path)[1]
j-parkes-nel commented 4 days ago

ditto here

ctx = ClientContext(sharepoint_base_url).with_user_credentials(my_info['username'], my_info['password'])
folder = ctx.web.folders.get_by_path("Shared Documents").get().execute_query()

folder
# web = ctx.web.get().execute_query()
# print('Connected to SharePoint: ',web.properties['Title'])

both of these don't work