search

shopware5 / SwagMediaS3

MIT License
22 stars 17 forks source link

Allowed memory size exhausted #15

Closed chrisribal closed 7 years ago

chrisribal commented 7 years ago

Our media folder has a size of about 50 GB. When calling the migration service with

php -f bin/console sw:media:migrate --from=local --to=s3

the script aborts after about 10 seconds with:

Allowed memory size of 4294967296 bytes exhausted

What can we do to migrate our media folder to s3?

EDIT: as you can see, currently i've set the PHP memory_limit to 4 GB

janbuecker commented 7 years ago

What Shopware version do you use for migrating to S3?

chrisribal commented 7 years ago

We're using 5.2.22. In the meantime we got the migration service running - we had to increase our memory_limit in the runtime to 16 GB. Apparently the service needs that space for us. As i could see the migration scripts load a list of all media files to migrate - which in our case are millions of files. Maybe there is better solution, like batch uploading or so?

janbuecker commented 7 years ago

Of course there is. We've updated the scripts to process the files one after another instead of processing all at once. The updated version will be published with 5.3.

In case you need the change now, you can backport it from here.

I think that the error is core related rather than the S3 adapter. Therefore I'll close this issue.

Thanks for the report @chrisribal.