storj / uplink

Storj network Go library
MIT License
122 stars 17 forks source link

Listing pending objects is broken #140

Closed mniewrzal closed 1 year ago

mniewrzal commented 1 year ago

I found quite old bug in listing code. When we are iterating over pending objects and we have more than 1000 (satellite side limit) we won't see more results. Iterator code is doing pages per max 1000 elements but it turns out that status field is not passed between pages. First ListObjects is having correct status with request but next request won't set it and satellite will get invalid status (0). It's working for committed objects because satellite have backward compatibility code to handle old uplinks which are not setting status at all and we are defaulting it to committed status. It looks that we are living with this issue since implementing multipart upload API.

func TestListUploads_Paging(t *testing.T) {
    testplanet.Run(t, testplanet.Config{
        SatelliteCount:   1,
        StorageNodeCount: 0,
        UplinkCount:      1,
    }, func(t *testing.T, ctx *testcontext.Context, planet *testplanet.Planet) {
        project, err := planet.Uplinks[0].OpenProject(ctx, planet.Satellites[0])
        require.NoError(t, err)
        defer ctx.Check(project.Close)

        createBucket(t, ctx, project, "testbucket")

        for i := 0; i < 10; i++ {
            _, err := project.BeginUpload(ctx, "testbucket", "object"+strconv.Itoa(i), nil)
            require.NoError(t, err)
        }

        listCtx := testuplink.WithListLimit(ctx, 3)
        list := project.ListUploads(listCtx, "testbucket", nil)

        items := 0
        for list.Next() {
            items++
        }
        require.Equal(t, 10, items)
    })
}
storj-gerrit[bot] commented 1 year ago

Change https://review.dev.storj.io/c/storj/uplink/+/9596 mentions this issue.