danlamanna / rivet

a tiny cli for syncing a directory with a girder instance
Apache License 2.0
6 stars 1 forks source link

Concurrent foldertree #2

Closed mgrauer closed 2 years ago

mgrauer commented 5 years ago

I made a hash of this: whitespace issues and commented out a big block of the full upload for ease of local testing. I'll end up cleaning up this branch for commit history and whitespace then will resubmit after we work out the path forward.

Obviously feel free to drop whatever QA you want on me, here is my starting point for discussion:

Whitespace: should we run go format on everything? I haven't done this consistently, but easy enough.

resources.go: added a new function to GetOrCreateFolder, assumes the parent folder exists, what is the right way to check/enforce this?

types.go: added GirderParentID and FolderChildren fields. FolderChildren only applies for Folder resources. Should I split these field additions out into perhaps a new Folder type that composes the Resource type with these fields and is used only for uploading?

upload.go: the meat of the work is in the addition of the function buildGirderFoldersConcurrent and a bit of extra supporting data in buildResourceMap. I may want to play with the structure in buildGirderFoldersConcurrent as I saw some other concurrency patterns in the Go Programming book, also/related I don't know how the overall concurrency behaves in the face of error given that the main goroutine relies on creating the a priori known number of folders before exiting the buildGirderFoldersConcurrent function.