Am I doing the gcp bucket set up right? keeps failing with EOF error

[emilaleksanteri]

Question Am I doing the gcp bucket set up right? keeps failing with EOF error, if I change the storage to be local file storage, then everything works perfect. My client is pretty much identical from the Uppy.js example for react Setup details Please provide following details, if applicable to your situation:

• Operating System: macOS
• Used tusd version: 2.4.0
• Used tusd data storage: Google Cloud Storage
• Used tusd configuration: Using the go library to have it on my server
• Used tus client library: tus-js-client, Uppy.js

Here is my current handler creation:

func FileUploadHandler() *tusd.UnroutedHandler {
    bucket := "sample-bucket"
    absPath, _ := filepath.Abs("./")
    gcpCredentials := filepath.Join(absPath, "config", "credentials.json")

    gcsService, err := gcsstore.NewGCSService(gcpCredentials)
    if err != nil {
        log.Fatalf("Unable to create gcsService: %s\n", err.Error())
    }

    locker := memorylocker.New()

    store := gcsstore.New(bucket, gcsService)
    store.ObjectPrefix = "store-products-csv"

    composer := tusd.NewStoreComposer()
    store.UseIn(composer)
    locker.UseIn(composer)

    handler, err := tusd.NewUnroutedHandler(tusd.Config{
        BasePath:              "/store-products-csv/",
        StoreComposer:         composer,
        NotifyCompleteUploads: true,
        NotifyCreatedUploads:  true,
        Cors: &tusd.CorsConfig{
            Disable: true,
        },
    })

    if err != nil {
        log.Fatalf("could not create tusd handler: %s\n", err.Error())
    }

    go func() {
        for {
            event := <-handler.CompleteUploads
            fmt.Printf("\nSuccessfully uploaded a file with an id: %v", event.Upload.ID)
            createdUpload := <-handler.CreatedUploads
            fmt.Printf("\nCreated an upload with an id: %v", createdUpload.Upload.ID)
        }
    }()

    return handler

}

routes set up with chi:

        r.Post("/store-products-csv/", fileRoutes.PostFile)
    r.Head("/store-products-csv/{id}", fileRoutes.HeadFile)
    r.Patch("/store-products-csv/{id}", fileRoutes.PatchFile)
    r.Get("/store-products-csv/{id}", fileRoutes.GetFile)

[Acconut]

That's odd. EOF usually means that the request got interrupted and thus tusd received the end of file (EOF) earlier than expected. Can you send us the tusd logs?

[emilaleksanteri]

That's odd. EOF usually means that the request got interrupted and thus tusd received the end of file (EOF) earlier than expected. Can you send us the tusd logs?

@Acconut this is what i see on my server logs though (changed the route path and file paths to test out something thats why its /files/ here

2024/04/30 11:58:47 "OPTIONS http://127.0.0.1:8081/files/ HTTP/1.1" from 127.0.0.1:51724 - 200 0B in 14.542µs
[tusd] 2024/04/30 11:58:47.722875 event="ResponseOutgoing" status="500" method="POST" path="/files/" error="EOF" requestId=""
2024/04/30 11:58:47 "POST http://127.0.0.1:8081/files/ HTTP/1.1" from 127.0.0.1:51724 - 500 4B in 9.0245ms
[tusd] 2024/04/30 11:58:47.727372 event="ResponseOutgoing" status="500" method="POST" path="/files/" error="EOF" requestId=""
2024/04/30 11:58:47 "POST http://127.0.0.1:8081/files/ HTTP/1.1" from 127.0.0.1:51724 - 500 4B in 3.38525ms
[tusd] 2024/04/30 11:58:50.745053 event="ResponseOutgoing" status="500" method="POST" path="/files/" error="EOF" requestId=""
2024/04/30 11:58:50 "POST http://127.0.0.1:8081/files/ HTTP/1.1" from 127.0.0.1:51724 - 500 4B in 12.250459ms

I dont see any cors issues but this is also my cors config in case it helps as well

r.Use(cors.Handler(cors.Options{
        AllowedOrigins: []string{"http://localhost:3000", "http://127.0.0.1:3000", "http://0.0.0.0:3000"},
        AllowedMethods: []string{"PUT", "PATCH", "GET", "POST", "OPTIONS", "DELETE", "HEAD"},
        AllowedHeaders: []string{"*"},
        ExposedHeaders: []string{
            "Upload-Offset",
            "Location",
            "Tus-Version",
            "Tus-Resumable",
            "Tus-Max-Size",
            "Tus-Extension",
            "Upload-Metadata",
            "Upload-Defer-Length",
            "Upload-Concat",
            "Upload-Offset",
            "Upload-Length",
        },
        AllowCredentials: true,
    }))

[Acconut]

Oh, you are getting EOF for POST requests? That's odd since usually no body is transferred for POST requests. What does your client code look like? Are you setting uploadDataDuringCreation?

[emilaleksanteri]

@Acconut so the client set up looks like this:

import { IslandContainer } from '@/components/Island';
import {
  Button,
  Grid,
  List,
  ListItem,
  ListItemIcon,
  ListItemText,
  Typography,
} from '@mui/material';
import { Folder } from '@mui/icons-material';
import React, { useState } from 'react';
import Uppy from '@uppy/core';
import Tus from '@uppy/tus';
import { Dashboard } from '@uppy/react';
import UppyDashboard from '@uppy/dashboard';

const STUBFILES = [
  {
    name: 'groceries-1.csv',
    date: new Date('04/26/2024').toJSON(),
  },
  {
    name: 'groceries-2.csv',
    date: new Date('04/26/2024').toJSON(),
  },
];

import '@uppy/core/dist/style.css';
import '@uppy/dashboard/dist/style.css';
import '@uppy/drag-drop/dist/style.css';
import '@uppy/file-input/dist/style.css';
import '@uppy/progress-bar/dist/style.css';

export default function Data() {
  const [showInlineDashboard, setShowInlineDashboard] = useState(false);
  const [uppy] = useState(() =>
    new Uppy({ debug: true, autoProceed: true }).use(UppyDashboard, { inline: true }).use(Tus, {
      endpoint: 'http://127.0.0.1:8081/store-products-csv/',
      retryDelays: [0, 3000],
      addRequestId: true,
      uploadDataDuringCreation: true,
    }),
  );

  return (
    <IslandContainer>
      <Grid>
        <Typography variant="h6">Uploaded product files</Typography>
        <List>
          {STUBFILES.map((file, idx) => (
            <ListItem key={idx}>
              <ListItemIcon>
                <Folder />
              </ListItemIcon>
              <ListItemText primary={file.name} secondary={file.date} />
            </ListItem>
          ))}
        </List>
        <label>
          <Button
            onClick={() => {
              setShowInlineDashboard(!showInlineDashboard);
            }}
          >
            Show Dashboard
          </Button>
        </label>
        {showInlineDashboard && (
          <Dashboard
            uppy={uppy}
            fileManagerSelectionType="files"
            metaFields={[{ id: 'name', name: 'Name', placeholder: 'File name' }]}
          />
        )}
      </Grid>
    </IslandContainer>
  );
}

Added the uploadDataDuringCreation as true now and the upload shows progress and then fails with EOF, logs show these errors now:

2024/05/02 10:55:45 "POST http://127.0.0.1:8081/store-products-csv/ HTTP/1.1" from 127.0.0.1:62786 - 500 31B in 4.202125ms
time=2024-05-02T10:55:48.504+01:00 level=ERROR msg=InternalServerError method=POST path=/store-products-csv/ requestId=9e8f9cb6-b57f-4471-ae11-b41c635cd419 message=EOF
time=2024-05-02T10:55:48.504+01:00 level=INFO msg=ResponseOutgoing method=POST path=/store-products-csv/ requestId=9e8f9cb6-b57f-4471-ae11-b41c635cd419 status=500 body="ERR_INTERNAL_SERVER_ERROR: EOF\n"
2024/05/02 10:55:48 "POST http://127.0.0.1:8081/store-products-csv/ HTTP/1.1" from 127.0.0.1:62786 - 500 31B in 14.373083ms

Looking at the bucket logs too, I can see that the .info gets created and its also taken from the bucket afterwards:

time=2024-05-02T10:06:06.437Z level=INFO msg="192.168.171.1 - - [02/May/2024:10:06:06 +0000] \"POST /upload/storage/v1/b/sample-bucket/o?alt=json&name=store-products-csv%2Fa9999f1e39fbc1cce2d37f4f790df38b.info&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1\" 200 912\n"
time=2024-05-02T10:06:06.439Z level=INFO msg="192.168.171.1 - - [02/May/2024:10:06:06 +0000] \"GET /sample-bucket/store-products-csv%2Fa9999f1e39fbc1cce2d37f4f790df38b.info HTTP/1.1\" 200 0\n"

[Acconut]

Did you have uploadDataDuringCreation set before? Does it change if you disable this?

[emilaleksanteri]

@Acconut Did not have it before, so if I disable it, the upload fails immediately with the same errors as before also a link to the repo im trying out the server on https://github.com/emilaleksanteri/tusd-demo-upload

[emilaleksanteri]

Here is also my browser network tab with the error when using uploadDataDuringCreation

[emilaleksanteri]

@Acconut after furhter investigation I noticed that the EOF error comes from this line in the gcsstore https://github.com/tus/tusd/blob/d24016a63c5d079eef8b1c44336d627ad074a97c/pkg/gcsstore/gcsstore.go#L157

So, the buf includes an invalid EOF byte at the end of it for some reason which causes the buf read to return an error although all of the .info data gets read into the buffer. Managed to get the file upload working fine if I ignore the r.Read(buf) error which is very weird Do you think that this is an error on gcs library side or on the tusd gcsservice?

[Acconut]

buf read to return an error although all of the .info data gets read into the buffer.

Thanks for looking into this, that's interesting! The EOF value in this case does not represent a real error, but rather that the file file was read from GCS. See the documentation for the Read method.

We can try fixing this by replacing

https://github.com/tus/tusd/blob/d24016a63c5d079eef8b1c44336d627ad074a97c/pkg/gcsstore/gcsstore.go#L156-L160

with

buf, err := io.ReadAll(r)
if err != nil {
    return info, err
}

ReadAll does not return EOF, so this error should not appear there. Could you give this a try and see if it helps?

[emilaleksanteri]

@Acconut that did the trick, want me to open a PR for this?

[Acconut]

That's great! Yes, please open a PR!