search

bensteUEM / ChurchToolsAPI

Collection of Python files to use the API for ChurchTools
6 stars 4 forks source link

Optimize Performance of file_upload #36

Closed bensteUEM closed 1 year ago

bensteUEM commented 1 year ago

Using SongbeamerQS uploading all files takes very long. looking at the log files each song attachment seems to be downloaded before overwritten.


2023-02-12 17:32:45,080 root       DEBUG    deleting old file before download
2023-02-12 17:32:45,172 urllib3.connectionpool DEBUG    https://elkw1610.krz.tools:443 "GET /api/files/song_arrangement/81 HTTP/1.1" 200 None
2023-02-12 17:32:45,263 urllib3.connectionpool DEBUG    https://elkw1610.krz.tools:443 "GET /?q=public/filedownload&id=216&filename=4374385c0f50be8728fbf784a958acfa.txt HTTP/1.1" 200 None
2023-02-12 17:32:45,350 urllib3.connectionpool DEBUG    https://elkw1610.krz.tools:443 "GET /?q=public/filedownload&id=222&filename=8188ac2327985931c00262a991462c32.pdf HTTP/1.1" 200 33158
2023-02-12 17:32:45,465 urllib3.connectionpool DEBUG    https://elkw1610.krz.tools:443 "DELETE /api/files/song_arrangement/81 HTTP/1.1" 204 0
2023-02-12 17:32:45,580 urllib3.connectionpool DEBUG    https://elkw1610.krz.tools:443 "POST /api/files/song_arrangement/81 HTTP/1.1" 200 None
2023-02-12 17:32:45,581 root       DEBUG    Upload successful {'data': [{'id': 7783, 'domainType': 'song_arrangement', 'domainId': '81', 'name': 'Jesus Erlöser der Welt - D.pdf', 'filename': 'b4120713e6096f5473303bec276e828d947fecfd54fc8d485bbf69eff96d7151', 'fileUrl': 'https://elkw1610.krz.tools/?q=public/filedownload&id=7783&filename=b4120713e6096f5473303bec276e828d947fecfd54fc8d485bbf69eff96d7151', 'imageUrl': None, 'relativeUrl': '?q=public/filedownload&id=7783&filename=b4120713e6096f5473303bec276e828d947fecfd54fc8d485bbf69eff96d7151', 'showOnlyWhenEditable': False, 'securityLevelId': None, 'type': 'file', 'size': 33158, 'additionalInfos': [], 'meta': {'createdDate': '2023-02-12T16:32:45Z', 'createdPerson': None, 'modifiedDate': '2023-02-12T16:32:45Z', 'modifiedPerson': None}}], 'meta': {'count': 1}}

This might be required because of an overwrite option - check that it's implemented the most efficient way when overwrite=True

bensteUEM commented 1 year ago

must be in file_delete method

  1. get file names
  2. for each file ... ` if current_name != filename_for_selective_delete: response = self.session.get(current_url) file_pathname = "tmp/{}{}".format(current_id, current_name) temp_file = open(file_path_name, "wb") temp_file.write(response.content) temp_file.close()

    downloads all files that should be kept

  3. response = self.session.delete(url=url)
bensteUEM commented 1 year ago

Problem is API use of /files/{domainType}/{domainIdentifier} // Deleting all files for arrangement vs. /files/{id} // only one file

method needs to check for file ID on delete instead of deleting everything

bensteUEM commented 1 year ago

IDs are part of the file list and can be accessed using [file[0] for file in online_files if file[1] == filename_for_selective_delete]

lines 354-366 maybe even further need adjustment..