Question on implementing offline first file upload with Brick

For the app I am building with Brick and Supabase, I have to implement file uploads in the upcoming weeks.

I already created a Brick model named FileMetadata which has the following properties:

id (String)
fileName (String)
downloadURL (String, nullable)
localPath (String, nullable)

The files should be uploaded to Supabase Storage.

When the client is online, this is no problem and I do it as follows:

upload the file to the Supabase storage via the Supabase SDK
get the download URL to the file
Construct the FileMetadata object using the download URL
upsert the FileMetadata with Brick

But when the client is offline, it becomes more tricky. What I have done so far is saving the file to the local app documents directory and construct the FileMetadata object, this time with the path to the local file instead of the download URL. Then I upsert the FileMetadata, which gets added to the offline request queue.

What is missing is uploading the File and adding its download URL to the FileMetadata object when the client comes back online and the request queue is processed. Before the FileMetadata gets sent to the remote database, I would have to upload the locally stored file and get its download URL to add it to the FileMetadata object.

In the issues and the docs, I could not find any hints or solutions on implementing offline first file upload with Brick. What I thought of is something like an beforeRemoteUpsert hook, which would get triggered, before the FileMetadata object gets sent to the remote database, in which I could upload the local file first, and set the download URL for the FileMetadata object.

I guess the easiest way would be to add the bytes of the file to the FileMetadata, but I would like to avoid storing files inside Databases.

@tshedor Do you have experience with using brick for file uploads or any ideas how to implement this? Thanks a lot as always!

@devj3ns So I've never had to do file upload with Brick. This is new territory. But, if I did, I would take an approach similar to what you've done by using the application documents directory.

Does your FileMetadata object have a table counterpart in your Supabase project or are you using the storage schema? When the client comes back online it could reconcile the difference between all files created locally in the last 7 days. The FileMetadata that exists locally but not in the remote would then be uploaded.

There's trigger from Brick that says "I'm back online, reprocess this." In the past, I've had a long running timer that checks for the queue length exceeding 0 for over 10 or so seconds. If it's over that 10 seconds, it would run a job like the reconciliation and reupload.

The other option you could do that feels slightly more hacky is overriding your repository's upsert method:

@override
Future<TModel extends BrickOfflineFirstWithRestModel> upsert<TModel>(instance, {query, repository}) async {
  if (TModel == FileMetadata) {
    // upload file logic
  }
  return super.upsert<TModel>(instance, query: query, repository: repository);
}

FWIW, my Brick app uploads files using:

a persistent queue I hacked together with Sembast for pending upload records;

local storage to hold the file until it is uploaded;

a background task that checks the queue, uploads the next file and removes the queued record.

Nothing special, but happy to share any of the pieces if that is helpful.

Thanks for your input on this!

@tshedor, yes, there is a database table in the Supabase Postgres public schema for the FileMetadata objects. In the Flutter app, this model uses the ConnectOfflineFirstWithRest annotation. So the offline capability and sync of the FileMetdata is already solved by Brick. The only missing piece is how to link the upload task to the FileMetadata object, which is stored in the offline queue when the client picks and adds files to the app while being offline.

Regarding the trigger: So you would check if the client is online (again) and upload all local files that have not been uploaded yet? With this approach, how would you handle the connection to the FileMetadata objects?

Regarding the upsert hack: To my understanding, the upsert method is only used when the client is online. When the client is offline, the upsert is transformed into an API Request containing the object to store in the remote database and saved inside the SQLite offline queue table. When going with this approach, I guess I had to also override the send method of the RestOfflineQueueClient to add the upload task before sending the FileMetada to the API while the offline request queue is processing. Am I getting this right?

@SubutaDan, thanks for sharing your solution. So you basically created a separate queue similar to Bricks offline request queue but for the file upload tasks. How do you handle the metadata of the files, do you have a database model that stores these? (With metadata I mean data like the user to which the file belongs to or a description).

Yes, I guess it is similar to the offline-first. queue, but mine is bare bones. The files that have to be uploaded are photos, each of which belongs to one of the items in a collection that is managed using offline first with REST. The name of the photo file is already written on the Brick record, along with the user id, tags, etc. The item that goes onto the queue is a small record that has the local file path and an identifier for the user to whom the file belongs.

On 26 August 2024 20:43:21 GMT+09:00, Jens Becker @.***> wrote:

Thanks for your input on this!

@tshedor, yes, there is a database table in the Supabase Postgres public schema for the FileMetadata objects. In the Flutter app, this model uses the ConnectOfflineFirstWithRest annotation. So the offline capability and sync of the FileMetdata is already solved by Brick. The only missing piece is how to link the upload task to the FileMetadata object, which is stored in the offline queue when the client picks and adds files to the app while being offline.

Regarding the trigger: So you would check if the client is online (again) and upload all local files that have not been uploaded yet? With this approach, how would you handle the connection to the FileMetadata objects?

Regarding the upsert hack: To my understanding, the upsert method is only used when the client is online. When the client is offline, the upsert is transformed into an API Request containing the object to store in the remote database and saved inside the SQLite offline queue table. When going with this approach, I guess I had to also override the send method of the RestOfflineQueueClient to add the upload task before sending the FileMetada to the API while the offline request queue is processing. Am I getting this right?

@SubutaDan, thanks for sharing your solution. So you basically created a separate queue similar to Bricks offline request queue but for the file upload tasks. How do you handle the metadata of the files, do you have a database model that stores these? (With metadata I mean data like the user to which the file belongs to or a description).

-- Reply to this email directly or view it on GitHub: https://github.com/GetDutchie/brick/issues/409#issuecomment-2310004522 You are receiving this because you were mentioned.

Message ID: @.***> -- Japan: +81 70 911 52405 US: +1 704 380 9253 UK: +44 7875 599 430

Regarding the trigger: So you would check if the client is online (again) and upload all local files that have not been uploaded yet?

Yes

With this approach, how would you handle the connection to the FileMetadata objects?

Is there a way that you can populate FileMetadata on the server instead of on the client? That way, the client would simply pull from the server instead of informing the server of a URL it's already created. I'm thinking you'd accomplish this with a function or with a SQL trigger.

If your FileMetadata was reliant on the server to create everything once you uploaded the file (say to a function endpoint), you could then reinvoke .get<FileMetadata> after a 200 response from the endpoint. Or you could short poll it, or you could use channels (though mind the cost, it escalates quickly).

What do you think of this route?

Regarding the upsert hack: To my understanding, the upsert method is only used when the client is online. When the client is offline, the upsert is transformed into an API Request containing the object to store in the remote database and saved inside the SQLite offline queue table. When going with this approach, I guess I had to also override the send method of the RestOfflineQueueClient to add the upload task before sending the FileMetada to the API while the offline request queue is processing. Am I getting this right?

You're right, I apologize for misleading on my suggestion. You would have to compose another http.Client (which you would do by passing that into RestProvider, the gzip mixin takes this approach). This isn't impossible, but it breaks encapsulation since you'd be invoking Repository within a client that is a dependency of Repository. At runtime it all works out, but it's not great architecture.

Unless your file uploads don't require Repository invocations, in which case you're doing something like what @SubutaDan is suggesting by having an external queue that's processing this stuff outside of Brick.

Good idea @tshedor, yes, I could create an endpoint which handles the file upload to Supabase Storage and the creation of the FileMetadata object in the database. This way, it would be easier to create a separate queuing system for the file uploads, as this separate queue would not depend on the FileMetadata objects, which would otherwise be stored inside Bricks offline queue.

I think I will implement a small prototype for both ways - the custom client and the separate queue - and then evaluate which way works better.

I will post an update if I have something working and sharable which others might profit from. Maybe this will help to add first party support for file uploads to brick in the future.

GetDutchie / brick

Question on implementing offline first file upload with Brick #409