Support handling millions of resources, such as entries, assets, users.

[videni]

In my recent experiment with Statamic, I found that Statamic Stache is not designed for millions of resources.

The statamic/eloquent-driver is not the final solution for this case, it simply can let you import/export data from file to database vice versa.

I know that it would take years for the community to support this case, so I decide to do some experiments myself, I create issue here to track things I am gona to do.

I don't think you want to handle millions of resource with flat files, so everything I talked below is based on the assumption - resources are kept in a database, I will try statamic/eloquent-driver first.

Todos

Don't loading all resources into memory

This is the biggest trouble I got at the moment.

• [ ] Don't paginate resources by loading all into memory For example, when you try to access the assets pages, the Statamic Stache will do pagination by loading all cached assets if exits into memory. otherwise it will try to cache all assets from asset container which can be remote dist, for example S3.
• [ ] Index data without load all into memory For example , the statamic:search:update command

[jonassiewertsen]

Statamic won't handle millions of resources without any adaptions. As written, a relational database also has its limits quickly (depending on different parameters).

Anyways: One custom feature with Statamic 3 is its abstractions: Handling all kinds of data relies on interfaces, so you can swap those out as you need and want them. Doing that is quite some work, but possible: https://statamic.com/blog/statamic-at-spiegel-scale

Maybe interesting as well: https://www.youtube.com/watch?v=yhiVC2GsJzs

Making it easier to handle millions of resources might be well appreciated by the community. Statamic did put a lot of work into it. so it's possible.

I am not sure if such a bulk issue does help, as I think it might be better if ideas were added to the ideas repo so that it can be discussed there. If issues need care, one problem per issue is the best way to go so that it can be discussed and PR's created regarding that issue. Having one issue might get crowded quickly, hard to track, and hard to discuss.

[jasonvarga]

Thank you for the issue, and for the explanation by Jonas.

I'm going to close this as these general "do it better" issues aren't really helpful and will just stay open forever. We of course want to be constantly improving.