Google Cloud SDK 322.0.0, Cloud Tools for Powershell (latest)
It appears that there is a consistent issue with "ls" or "dir" when trying to list objects of a folder (eg. PS gs:\bucketX> ls folderB) when there are over 1,000 objects stored prior to the target folder.
Sample output:
PS gs:\bucketTest> dir
ParentPath: bucketTest
Name Size ContentType TimeCreated Updated
---- ---- ----------- ----------- -------
A
B
C
PS gs:\bucketTest> dir B
dir : Cannot find path 'bucketTest\B' because it does not exist.
At line:1 char:1
+ dir B
+ ~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (bucketTest\B:String) [Get-ChildItem], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand
Debug/Troubleshoot details:
I managed to execute the tool in verbose mode via Visual Studio and tracked where the problem is in the codes. The first logic entry [1], and next the issue is caused by the this line [2] as it constantly returns "false".
_objectMap appears to be an array holding the result of the call to storage.objects.list REST API [3] which limits the object arrays up to 1,000 maximum
As the objects array are alphabetically ordered, this appears to be the reason why if there are 1,000 objects prior the target folder, the _objectMap array would not contain the target folder object hence the .ContainsKey(objectName) always return "false"
Reproduction steps:
1) Create bucketTest with folder "A", "B", and "C"
2) Upload 1,010 objects to folder "A"
3) Open Powershell (assuming necessary configs are done)
4) Enter cd gs:\
5) PS gs:\> cd bucketTest
6) PS gs:\bucketTest> ls B (ItemNotFoundException expected to occur here)
Documentation:
Summary:
PS gs:\bucketX> ls folderB) when there are over 1,000 objects stored prior to the target folder.Debug/Troubleshoot details:
_objectMapappears to be an array holding the result of the call tostorage.objects.listREST API [3] which limits the object arrays up to 1,000 maximum_objectMaparray would not contain the target folder object hence the.ContainsKey(objectName)always return "false"Reproduction steps:
1) Create bucketTest with folder "A", "B", and "C" 2) Upload 1,010 objects to folder "A" 3) Open Powershell (assuming necessary configs are done) 4) Enter
cd gs:\5)PS gs:\> cd bucketTest6)PS gs:\bucketTest> ls B(ItemNotFoundException expected to occur here)References:
[1] https://github.com/GoogleCloudPlatform/google-cloud-powershell/blob/b0549f98b2a150830ff6ddf10c66c3252df7626d/Google.PowerShell/Provider/GoogleCloudStorageProvider.cs#L390
[2] https://github.com/GoogleCloudPlatform/google-cloud-powershell/blob/b0549f98b2a150830ff6ddf10c66c3252df7626d/Google.PowerShell/Provider/BucketModel.cs#L108
[3] https://cloud.google.com/storage/docs/json_api/v1/objects/list#maxResults
Current Workaround
The following works fine and offer similar object listing results for Cloud Storage bucket folder: