Add option to preserve original file name

[botenvouwer]

Is your feature request related to a problem? Please describe. When uploading to a backend like azure blob storage it is not possible to change the key. The landscape using the files we upload use original file names. We like to use tus to upload very large files like 200gb. Now when we want to rename the key we need to copy the entire file again so we have the original file key. The uniqueness of the file is already taken care of based on the upload key we want to provide.

Describe the solution you'd like An option to preserve or to configure a custom destination key for usage with blob storage or s3 like backends. The uniqueness of a file name is then taken care off by the user who provides a key. When a key already exists the file could be overwritten. When the meta data already exists the upload can be resumed etc.

Describe alternatives you've considered Right now we are forced to upload the files to a static location and copy them to final destination with original filename as first step of our ETL workflow.

Can you provide help with implementing this feature? We have some hours available from our employer (Dutch government Geological Survey agency: Kadaster) and we have colleagues working with go. We could sure spare some time to help implementing.

[Acconut]

tusd v2, which was released two weeks ago, includes a feature to customize the upload ID (and hence the upload location) on the server-side during the initial stages of the upload. You can read more at https://github.com/tus/tusd/blob/main/docs/hooks.md, in particular the pre-create hook is relevant for your question.

[botenvouwer]

tusd v2, which was released two weeks ago, includes a feature to customize the upload ID (and hence the upload location) on the server-side during the initial stages of the upload. You can read more at https://github.com/tus/tusd/blob/main/docs/hooks.md, in particular the pre-create hook is relevant for your question.

Great! I'm looking into it right now. We like the option to use TUSD programmatically like described here. How would you implement the pre-create hook in this case? Or is this not supported?

[Acconut]

You can use the Config.PreUploadCreateCallback field to specify a callback which acts similarly to the pre-create hook. Effectively, that is how hooks are implemented as well.

[botenvouwer]

@Acconut Thanks for your tip! We implemented a test case like this:

func main() {

    azureConfig := &azurestore.AzConfig{
        AccountName:   "localazurite",
        AccountKey:    "localazurite",
        ContainerName: "test",
        Endpoint:      "http://127.0.0.1:10000/localazurite",
    }

    azureService, err := azurestore.NewAzureService(azureConfig)
    azureStore := azurestore.New(azureService)

    composer := handler.NewStoreComposer()
    azureStore.UseIn(composer)

    handler, err := handler.NewHandler(handler.Config{
        BasePath:                "/upload/",
        StoreComposer:           composer,
        NotifyCompleteUploads:   true,
        NotifyUploadProgress:    true,
        NotifyCreatedUploads:    true,
        PreUploadCreateCallback: preupload,
    })

    if err != nil {
        panic(fmt.Errorf("Unable to create handler: %s", err))
    }

    go func() {
        for {
            event := <-handler.CreatedUploads
            fmt.Printf("Upload %s Started\n", event.Upload.ID)
        }
    }()

    go func() {
        for {
            event := <-handler.CompleteUploads
            fmt.Printf("Upload %s Finished\n", event.Upload.ID)
        }
    }()

    http.Handle("/upload/", http.StripPrefix("/upload/", handler))
    err = http.ListenAndServe(":8080", nil)
    if err != nil {
        panic(fmt.Errorf("Unable to listen: %s", err))
    }

    //http.Handle("/files/", http.StripPrefix("/files/", fileService))
    //err = http.ListenAndServe(":8080", nil)
    //if err != nil {
    //  panic(fmt.Errorf("Unable to listen: %s", err))
    //}
}

func preupload(hook handler.HookEvent) (handler.HTTPResponse, handler.FileInfoChanges, error) {
    httpResponse := handler.HTTPResponse{}

    fileInfo := handler.FileInfoChanges{
        ID: "bla_test2",
        Storage: map[string]string{
            "Type":      "azurestore",
            "Container": "test",
            "Key":       "bla/bla_test2",
        },
    }

    return httpResponse, fileInfo, nil
}

This is great and extensible!

[Acconut]

That looks correct, I am glad you figured it out :) I will close this issue then as it seems you question was answered. Feel free to continue here or open a new issue if there are other problems.