search

thephpleague / flysystem-rackspace

Flysystem Adapter for Rackspace
37 stars 28 forks source link

Circumvent Rackspace's 10,000 object return limit #5

Closed NordomWhistleklik closed 9 years ago

NordomWhistleklik commented 9 years ago

The Rackspace $container->objetList() call only returns 10,000 results max. In order to get around this, you need to make multiple calls, passing in the last object received as a marker.

frankdejonge commented 9 years ago

@NordomWhistleklik while the tests are not finished, I'm pretty sure they'll fail or lose coverage somewhere here. If you want I can re-implement this with tests or you can add tests for this. Might I suggest to extract everything in the while loop out into a separate method as well? I can do this after the PR to if you have difficulties doing this.

NordomWhistleklik commented 9 years ago

Sorry, didn't mean to submit a buggy or incomplete PR. I'm fairly new to Github collaboration. Feel free to re-implement with tests and however see fit. I was just banging my head against a wall trying to figure out what was going wrong and I was happy I found a solution I could share.

frankdejonge commented 9 years ago

@NordomWhistleklik no worries. I've implemented it in a slight different way, with tests. Can you verify this fix?

NordomWhistleklik commented 9 years ago

I don't believe your change (1104887e3bcbda5c9f4c786c07a216046490ba27) works. After a couple seconds my server runs out of memory. If I'm reading it right, it's just requesting the same objectList over and over again (the first 10,000 objects in my case) and appending them together forever. The $objectList->count() will never equal 0 without passing each objectList() call the "marker" parameter.

frankdejonge commented 9 years ago

@NordomWhistleklik I just woke up, knowing I committed an eternal loop. Fix coming up.

frankdejonge commented 9 years ago

@NordomWhistleklik the fix was pushed a couple hours ago :)

NordomWhistleklik commented 9 years ago

Awesome! Thanks for doing all that! It's listing all my files now and the code looks a lot cleaner than mine did :)

frankdejonge commented 9 years ago

@NordomWhistleklik thank you :) I'll tag a release so you don't have to rely on dev-master.

frankdejonge commented 9 years ago

1.0.1 has been tagged