Nightly failure while replacing file with zero-byte file

[PrajwolAmatya]

Describe the bug

Nightly failure on scenario related to replacing a file with zero-byte file.

@issue-8003
  Scenario: replace a file inside a project space with zero-byte file                                                                                  # /drone/src/tests/acceptance/features/apiSpaces/tusUpload.feature:120
    Given using spaces DAV path                                                                                                                        # FeatureContext::usingOldOrNewDavPath()
    And the administrator has assigned the role "Space Admin" to user "Alice" using the Graph API                                                      # GraphContext::theAdministratorHasGivenTheRoleUsingTheGraphApi()
    And user "Alice" has created a space "new-space" with the default quota using the Graph API                                                        # SpacesContext::theUserHasCreatedASpaceByDefaultUsingTheGraphApi()
    And user "Alice" has uploaded a file inside space "new-space" with content "This is TUS upload" to "textfile.txt"                                  # SpacesContext::userHasUploadedFile()
    When user "Alice" uploads a file from "filesForUpload/zerobyte.txt" to "textfile.txt" via TUS inside of the space "new-space" using the WebDAV API # SpacesTUSContext::userUploadsAFileViaTusInsideOfTheSpaceUsingTheWebdavApi()
      Server error: `POST https://ocis-server:9200/remote.php/dav/spaces/e0ee475d-8b1b-4035-a007-19c40dcbe4a0$b1f54481-568c-4e6b-93c8-db91379774c4/` resulted in a `500 Internal Server Error` response (GuzzleHttp\Exception\ServerException)
    Then for user "Alice" the content of the file "textfile.txt" of the space "new-space" should be ""                                                 # SpacesContext::checkFileContent()

The test seems to be flaky, and there was an issue reported here: https://github.com/owncloud/ocis/issues/8003 regarding replacing a file, which returned 500 status code.

Build: https://drone.owncloud.com/owncloud/ocis/36918/19/5 ocis log:

{"level":"error","pid":2589,"pid":2589,"error":"failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /root/.ocis/storage/users/uploads/c4931b01-09e7-4329-9e2f-43ba2249771d: no such file or directory","time":"2024-07-10T00:48:13Z","caller":"/drone/src/vendor/github.com/cs3org/reva/v2/internal/grpc/services/storageprovider/storageprovider.go:415","message":"failed to upload"}
{"level":"error","service":"storage-users","pkg":"rgrpc","traceid":"23c32f2183c570370e6758683fcc3602","error":"failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /root/.ocis/storage/users/uploads/c4931b01-09e7-4329-9e2f-43ba2249771d: no such file or directory","status":{"code":15,"message":"error getting upload id: failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /root/.ocis/storage/users/uploads/c4931b01-09e7-4329-9e2f-43ba2249771d: no such file or directory","trace":"23c32f2183c570370e6758683fcc3602"},"time":"2024-07-10T00:48:13Z","message":"failed to initiate upload"}
{"level":"error","service":"ocdav","name":"com.owncloud.web.ocdav","traceid":"23c32f2183c570370e6758683fcc3602","request-id":"e9341c4f6410/e5mDpZ3pyT-000286","spaceid":"e0ee475d-8b1b-4035-a007-19c40dcbe4a0$b1f54481-568c-4e6b-93c8-db91379774c4","path":"/","filename":"textfile.txt","status":{"code":15,"message":"error getting upload id: failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /root/.ocis/storage/users/uploads/c4931b01-09e7-4329-9e2f-43ba2249771d: no such file or directory","trace":"23c32f2183c570370e6758683fcc3602"},"code":500,"time":"2024-07-10T00:48:13Z","message":"Internal Server Error"}

[saw-jan]

Most likely related API test failures:

• https://drone.owncloud.com/owncloud/ocis/37182/50/5

  Scenario: user updates a file inside a project space with empty content                                            # /drone/src/tests/acceptance/features/coreApiWebdavUpload/uploadFile.feature:351
  Given using spaces DAV path                                                                                      # FeatureContext::usingOldOrNewDavPath()
  And the administrator has assigned the role "Space Admin" to user "Alice" using the Graph API                    # GraphContext::theAdministratorHasGivenTheRoleUsingTheGraphApi()
  And user "Alice" has created a space "new-space" with the default quota using the Graph API                      # SpacesContext::theUserHasCreatedASpaceByDefaultUsingTheGraphApi()
  And user "Alice" has uploaded a file inside space "new-space" with content "file with content" to "textfile.txt" # SpacesContext::userHasUploadedFile()
  When user "Alice" uploads a file inside space "new-space" with content "" to "textfile.txt" using the WebDAV API # SpacesContext::theUserUploadsAFileToSpace()
  Then the HTTP status code should be "204"                                                                        # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    HTTP status code 500 is not the expected value 204
    Failed asserting that 500 matches expected '204'.

• https://drone.owncloud.com/owncloud/ocis/37180/49/5

  Scenario Outline: copy a file of size zero byte                                                      # /drone/src/tests/acceptance/features/coreApiWebdavProperties/copyFile.feature:558
  Given using <dav-path-version> DAV path                                                            # FeatureContext::usingOldOrNewDavPath()
  And user "Alice" has uploaded file "filesForUpload/zerobyte.txt" to "/zerobyte.txt"                # FeatureContext::userHasUploadedAFileTo()
  And user "Alice" has created folder "/testZeroByte"                                                # FeatureContext::userHasCreatedFolder()
  When user "Alice" copies file "/zerobyte.txt" to "/testZeroByte/zerobyte.txt" using the WebDAV API # FeatureContext::userCopiesFileUsingTheAPI()
  Then the HTTP status code should be "201"                                                          # FeatureContext::thenTheHTTPStatusCodeShouldBe()
  And as "Alice" file "/testZeroByte/zerobyte.txt" should exist                                      # FeatureContext::asFileOrFolderShouldExist()
  And as "Alice" file "/zerobyte.txt" should exist                                                   # FeatureContext::asFileOrFolderShouldExist()
  
  Examples:
    | dav-path-version |
    | old              |
      Failed step: Then the HTTP status code should be "201"
      HTTP status code 500 is not the expected value 201
      Failed asserting that 500 matches expected '201'.

• https://drone.owncloud.com/owncloud/ocis/37147/50/5

• https://drone.owncloud.com/owncloud/ocis/37202/53/5

  Scenario Outline: user updates a shared file with empty content                                                # /drone/src/tests/acceptance/features/coreApiWebdavUpload/uploadFile.feature:330
  Given using <dav-path-version> DAV path                                                                      # FeatureContext::usingOldOrNewDavPath()
  And user "Brian" has been created with default attributes and without skeleton files                         # FeatureContext::userHasBeenCreatedWithDefaultAttributes()
  And user "Alice" has uploaded file with content "file with content" to "/textfile.txt"                       # FeatureContext::userHasUploadedAFileWithContentTo()
  And user "Alice" has sent the following resource share invitation:                                           # SharingNgContext::userHasSentTheFollowingResourceShareInvitation()
    | resource        | textfile.txt |
    | space           | Personal     |
    | sharee          | Brian        |
    | shareType       | user         |
    | permissionsRole | File Editor  |
  When user "Brian" uploads file with content "" to shared resource "Shares/textfile.txt" using the WebDAV API # FeatureContext::userUploadsFileWithContentSharedResourceToUsingTheWebdavApi()
  Then the HTTP status code should be "204"                                                                    # FeatureContext::thenTheHTTPStatusCodeShouldBe()
  And for user "Brian" the content of the file "/test.txt" of the space "Shares" should be ""                  # SpacesContext::checkFileContent()
  And the content of file "/textfile.txt" for user "Alice" should be ""                                        # FeatureContext::contentOfFileForUserShouldBe()
  
  Examples:
    | dav-path-version |
    | old              |
    | new              |
      Failed step: Then the HTTP status code should be "204"
      HTTP status code 500 is not the expected value 204
      Failed asserting that 500 matches expected '204'.

[saw-jan]

Pattern observed: Tests related to uploading zero-byte file fail

[saw-jan]

ocis log:

{
  "level": "error",
  "spaceid": "622a3b63-98a0-44b4-893f-69d44cf7dc33",
  "nodeid": "90df2eed-c368-4ce0-bc6e-ee0bf43c5b92",
  "error": "failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /var/lib/ocis/storage/users/uploads/6036af4f-388c-49aa-a40c-5251d3dbe63a: no such file or directory",
  "time": "2024-07-18T07:00:22Z",
  "message": "could not finalize upload"
}
{
  "level": "error",
  "event": "PostprocessingFinished",
  "uploadid": "30e2da95-f7c3-4b8a-8579-ba4823ed00eb",
  "error": "upload not found",
  "time": "2024-07-18T07:00:28Z",
  "message": "Failed to get upload"
}
{
  "level": "error",
  "pid": 1,
  "error": "failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /var/lib/ocis/storage/users/uploads/c10d8576-0bf3-4ef2-875c-5a0453fb8a97: no such file or directory",
  "time": "2024-07-18T07:00:35Z",
  "caller": "github.com/cs3org/reva/v2@v2.21.1-0.20240716111044-11ee452e19a7/internal/grpc/services/storageprovider/storageprovider.go:416",
  "message": "failed to upload"
}
{
  "level": "error",
  "service": "storage-users",
  "pkg": "rgrpc",
  "traceid": "5f88ce50846e6ef8e28e1dd3642bf392",
  "error": "failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /var/lib/ocis/storage/users/uploads/c10d8576-0bf3-4ef2-875c-5a0453fb8a97: no such file or directory",
  "status": {
    "code": 15,
    "message": "error getting upload id: failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /var/lib/ocis/storage/users/uploads/c10d8576-0bf3-4ef2-875c-5a0453fb8a97: no such file or directory",
    "trace": "5f88ce50846e6ef8e28e1dd3642bf392"
  },
  "time": "2024-07-18T07:00:35Z",
  "message": "failed to initiate upload"
}
{
  "level": "error",
  "service": "ocdav",
  "name": "com.owncloud.web.ocdav",
  "traceid": "5f88ce50846e6ef8e28e1dd3642bf392",
  "request-id": "1a6f85253a39/WsFJGOztKZ-008222",
  "path": "/users/88855926-eb79-4144-b125-d2e9c794f860/Shares/textfile.txt",
  "status": {
    "code": 15,
    "message": "error getting upload id: failed to upload file to blobstore: Decomposedfs: oCIS blobstore: Can not open source file to upload: open /var/lib/ocis/storage/users/uploads/c10d8576-0bf3-4ef2-875c-5a0453fb8a97: no such file or directory",
    "trace": "5f88ce50846e6ef8e28e1dd3642bf392"
  },
  "code": 500,
  "time": "2024-07-18T07:00:35Z",
  "message": "Internal Server Error"
}

What could go wrong while uploading the zero-byte file?

CC @micbar @kobergj @2403905

[kobergj]

I think there is a problem when uploading 0 byte files as new version of an existing file. I can reproduce an error message like:

{
  "level": "error",
  "event": "PostprocessingFinished",
  "uploadid": "3a262178-9d92-4b1c-8d73-04318119d8b2",
  "error": "upload not found",
  "time": "2024-07-22T11:54:41+02:00",
  "message": "Failed to get upload"
}

However the upload succeeds and I see none of the other logs mentioned.

[kobergj]

Should be fixed with above PR. Please reopen if this issue persists after the PR was merged