search

vgrem / Office365-REST-Python-Client

Microsoft 365 & Microsoft Graph Library for Python
MIT License
1.33k stars 334 forks source link

URL limit hit when preforming moveto and copyto #319

Open thomasfrederikhoeck opened 3 years ago

thomasfrederikhoeck commented 3 years ago

SharePoint API has a limit to the length of the URL which can be used for the API at 260 characters. This is not normally a problem but if you are performing MoveTo or CopyTo, where the files are located deep in the folders the limits can easily be hit.

Right now the relative URL is used to signal which file to move but if the library switched to using the unique file id then this probelm would be much smaller (see https://stackoverflow.com/questions/36172818/url-length-limitation-in-sharepoint-online-rest).

thomasfrederikhoeck commented 3 years ago

@trinebrockhoff

Albarion commented 3 years ago

Hello, I'm also running into this problem, will there be a fix or is a workaround available ?

dvinesett commented 2 years ago

I created a workaround using some of Microsoft's documentation on moveto. I think copyto could be swapped in easily but I did not verify. It uses RequestOptions to make the REST query manually. I tried to use 'public' methods from the library whenever possible.

# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.path_builder import ODataPathBuilder

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataPathBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)

EDIT: I updated the workaround per changes in the commit: https://github.com/vgrem/Office365-REST-Python-Client/commit/693520b24853b8e4321eaf28ba21034601dc7659

dvinesett commented 2 years ago

I updated my workaround to no longer use moveto() from office365.sharepoint.files.file. Instead, I just hardcode the method_name when building the ODataPath. Calling moveto was unnecessary and had the side affect of adding a moveto query into the context's _queries list.

mwmg commented 2 years ago

I'm also running into this problem. Unfortunately the workaround returns an error 403 for me, with the following message: '{"error":{"code":"-2130575251, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The security validation for this page is invalid and might be corrupted. Please use your web browser\'s Back button to try your operation again."}}}'

I have authenticated using the following method and this has worked for me with built-in queries until now: CTX= (ClientContext(site_url).with_credentials(UserCredential(username, password)))

Do you have any suggestions on how to fix this?

oliaboo commented 2 years ago

I'm also running into this problem. Unfortunately the workaround returns an error 403 for me, with the following message: '{"error":{"code":"-2130575251, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The security validation for this page is invalid and might be corrupted. Please use your web browser\'s Back button to try your operation again."}}}'

I have authenticated using the following method and this has worked for me with built-in queries until now: CTX= (ClientContext(site_url).with_credentials(UserCredential(username, password)))

Do you have any suggestions on how to fix this?

I've faced the same issue. The fix is: CTX.ensure_form_digest(request) this will add X-RequestDigest to the header.

mwmg commented 2 years ago

Thank you, that worked!

dvinesett commented 2 years ago

It looks like a recent commit broke my workaround. The updated README was helpful in correcting its methodology. I've fixed it on the original comment: https://github.com/vgrem/Office365-REST-Python-Client/issues/319#issuecomment-1112892198

Note-- the change was committed after the problem reported above: https://github.com/vgrem/Office365-REST-Python-Client/issues/319#issuecomment-1216386742

skyashin commented 1 year ago

I created a workaround using some of Microsoft's documentation on moveto. I think copyto could be swapped in easily but I did not verify. It uses RequestOptions to make the REST query manually. I tried to use 'public' methods from the library whenever possible.

# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.path_builder import ODataPathBuilder

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataPathBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)

EDIT: I updated the workaround per changes in the commit: 693520b

Can you please share code to copy instead of move

skyashin commented 1 year ago

Can you please share the code to COPY instead of MOVE using ODATA method?

FedericoRaimondi commented 1 year ago

I created a workaround using some of Microsoft's documentation on moveto. I think copyto could be swapped in easily but I did not verify. It uses RequestOptions to make the REST query manually. I tried to use 'public' methods from the library whenever possible.

# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.path_builder import ODataPathBuilder

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataPathBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)

EDIT: I updated the workaround per changes in the commit: 693520b

Hello, thanks for the workaround! I just needed to update following the renaming of ODataPathBuilder (commit: 74f95a9):


# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.url_builder import ODataUrlBuilder
# from office365.runtime.odata.path_builder import ODataPathBuilder # old

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataUrlBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)
Necolanch commented 3 months ago

I created a workaround using some of Microsoft's documentation on moveto. I think copyto could be swapped in easily but I did not verify. It uses RequestOptions to make the REST query manually. I tried to use 'public' methods from the library whenever possible.

# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.path_builder import ODataPathBuilder

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataPathBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)

EDIT: I updated the workaround per changes in the commit: 693520b

Hello, thanks for the workaround! I just needed to update following the renaming of ODataPathBuilder (commit: 74f95a9):

# additional imports
from office365.runtime.http.request_options import RequestOptions
from office365.runtime.odata.url_builder import ODataUrlBuilder
# from office365.runtime.odata.path_builder import ODataPathBuilder # old

# ...

file = CTX.web.get_file_by_server_relative_url(source_filepath).get().execute_query()
file_id = file.unique_id

get_file_query_string = "getFileById('{}')".format(file_id)

moveto_query_params = {'newurl': future_filepath, 'flags': 1}
moveto_query_string = ODataUrlBuilder.build('moveto', moveto_query_params)

moveto_url = '/'.join([CTX.service_root_url(), 'web', get_file_query_string, moveto_query_string])
request = RequestOptions(moveto_url)
request.method = 'POST'
CTX.pending_request().execute_request_direct(request)

minus any sensitive information, what ends up being the value for the future_filepath variable being used for the 'newurl' argument in the moveto_query_params dictionary/object?

Additionally, as of the date of this comment, the library files do not contain the path builder or url builder, resulting in a module not found error. so i am employing the help of GitHub Copilot to help with creating it. Will update with my produced, working value.

spirosbond commented 3 months ago

I struggled with the same issue. I ended up using the download() and upload() functions to get around the problem of too long URL with copyto().