Add support for S3 buckets for userdata

[greenscientist]

Instead of relying on file storage for user data, we could use S3 buckets instead.

This could be cheaper and more space flexible (we could use only the required space instead of have to allocate space before hand), depending on the infrastrure.

[greenscientist]

https://www.npmjs.com/package/@flystorage/file-storage (https://flystorage.dev/api/) is a potential package to use, as it can abstract storing to disk or on S3 (even others)

Is there better alternative packages ?

[greenscientist]

Comments from a git grep userDataD

packages/chaire-lib-backend/src/services/users/users.ts: const absoluteUserDir = ${directoryManager.userDataDirectory}/${userId}; => Quota usage calculation, can be wrapped differently

packages/chaire-lib-backend/src/utils/filesystem/directoryManager.ts: private _userDataDirectory: string; packages/chaire-lib-backend/src/utils/filesystem/directoryManager.ts: this._userDataDirectory = path.normalize(`${this._projectDirec> packages/chaire-lib-backend/src/utils/filesystem/directoryManager.ts: get userDataDirectory() { packages/chaire-lib-backend/src/utils/filesystem/directoryManager.ts: return this._userDataDirectory; => Will need rework, to return the right information

packages/transition-backend/src/api/tests/gtfs.routes.test.ts: expect(mockGtfsImport).toHaveBeenCalledWith(`${directoryManag> packages/transition-backend/src/api/tests/gtfs.routes.test.ts: expect(mockedGtfsExport).toHaveBeenCalledWith(exportData.sele> packages/transition-backend/src/api/tests/gtfs.routes.test.ts: expect(mockedGtfsExport).toHaveBeenCalledWith(exportData.sele> => Tests, probably just need to be adapted to the new reality

packages/transition-backend/src/api/gtfs.socketRoutes.ts: const absoluteUserDir = ${directoryManager.userDataDirectory}/${userId}; => Per user "scratch space". We don't need to really save the GTFS => Upload seems elsewhere...

packages/transition-backend/src/api/services.socketRoutes.ts: filepath: ${directoryManager.userDataDirectory}/$> packages/transition-backend/src/api/services.socketRoutes.ts: input:${directoryManager.userDataDirectory}/${us> => Refer to the path where the last batch*.csv file was uploaded. => Again, this could be a scratch space, while we process.

packages/transition-backend/src/api/uploads.socketRoutes.ts: const absoluteUserDir = ${directoryManager.userDataDirectory}/${userId}; => This use the ancien SocketIOFile, not maintained since 4-5 years ago, so we might need to file a whole replacement for it.

packages/transition-backend/src/serverApp.ts: if (fileManager.fileExistsAbsolute(${directoryManager.userDataDirectory}/${userId}/ex> packages/transition-backend/src/serverApp.ts: res.sendFile(path.join(directoryManager.userDataDirectory, String(userId), 'export> packages/transition-backend/src/serverApp.ts: if (fileManager.fileExistsAbsolute(${directoryManager.userDataDirectory}/${userId}/${> packages/transition-backend/src/serverApp.ts: res.sendFile(path.join(directoryManager.userDataDirectory, String(userId), String(> => Express Upload, can replace the res.sendFile by a res.send with a Buffer

packages/transition-backend/src/services/executableJob/ExecutableJob.ts: return `${directoryManager.userDataDirectory}/${userId !== > => This path will be the more complicated, as goes a few level deep. => It's also maybe the only one for now that we really want to save to S3. => At least seems somewhat well contained => All access via getJobFileDirectory() will need to be checked

packages/transition-backend/src/services/transitRouting/tests/TrAccessibilityMapBatch.test.ts:const absoluteDir = ${directoryManager.u> packages/transition-backend/src/services/transitRouting/__tests__/TrRoutingBatch.test.ts:const absoluteDir =${directoryManager.userDataDi> packages/transition-backend/src/services/transitRouting/tests/TrRoutingBatchResult.test.ts:const absoluteDir = `${directoryManager.user> => Related to the ExecutableJob, those test will need to be adapted and highlight how deep the ExecutableJob does