google-research / CVD-paper-mobile-camera-example

Apache License 2.0
6 stars 4 forks source link

Batch Uploads: Streamline uploading #56

Closed MJ1998 closed 6 months ago

MJ1998 commented 6 months ago

This pull request introduces enhancements to the synchronization logic for UploadRequest records, aiming to streamline the process and improve code clarity. The core logic of the synchronization is outlined in the synchronize function, which returns a Flow.

The core logic is :-

suspend fun synchronize(): Flow<SyncUploadState> = flow {
  emit(SyncUploadState.Started)
  var uploadRequestList = uploadRequestFetcher.fetchForUpload()
  while (uploadRequestList.isNotEmpty()) {
    uploader
      .upload(uploadRequestList)
      .onEach { uploadResultProcessor.process(it) }
      .runningFold(initialSyncUploadState, ::calculateSyncUploadState)
      .drop(1)
      .collect { emit(it) }
    uploadRequestList = uploadRequestFetcher.fetchForUpload()
  }
  emit(SyncUploadState.Completed)
}

The process begins with emitting a SyncUploadState.Started signal. The uploadRequestFetcher is then employed to fetch a list of UploadRequest records for upload. The loop continues until no records are left to process.

Within the loop:

  1. The uploader is utilized to upload the current batch of uploadRequestList.
  2. The uploadResultProcessor processes each upload result.
  3. The runningFold function is employed to calculate the SyncUploadState based on the previous "InProgress" states, with the initial state provided by initialSyncUploadState.
  4. The first element of the folded result is dropped, and the subsequent SyncUploadState is emitted through the flow.

Additionally, variable names have been adjusted to better reflect the dynamic nature of totalRequests during upload in progress.

The overall result is a more streamlined and efficient synchronization process for managing UploadRequest records, enhancing code readability and addressing the mentioned issue.