search

un7c0rn / stopthevirus

Objective: Design a global scale social game to help inspire millions of Millennial and Gen-Z individuals across the world to engage in social distancing in order to stop the spread of the COVID-19 virus.
MIT License
7 stars 2 forks source link

Create large data set for testing purposes #46

Open bernardbaker opened 4 years ago

bernardbaker commented 4 years ago

@un7c0rn I need a large data set for testing purposes.

Can you write a 🐍script that I could run?

The reason for this is because I didn't implement from concurrent.futures import ThreadPoolExecutor as you did in the python script firestore.py.

So we need to run tests to see how well the API processes cope. And measure usage, per user with different sized data sets.

E.g. 1 x User makes a request of the API which processes 1,000 entries in the Firestore DB.

Some information on the Netlify quota for serverless functions:

By default, all serverless functions are deployed with:

I know that Firestore is blazingly fast at processing. But it would be good to have a dashboard if one isn't already available.

🌞

I think this could help web workers. I'll look at it tomorrow.

un7c0rn commented 4 years ago

@bernardbaker Sounds good can you provide a quick example of the format you need the test data to be in? I'm a little concerned about the Netlify quota. I'm assuming we can scale up requests for a fee as necessary? If we use GCP to hose the JS lambda functions we get 2M / month free: https://cloud.google.com/functions/pricing

That said, I'm also in favor of getting a working end-to-end testable MVP done and worrying about pricing optimizations as a second step.

bernardbaker commented 4 years ago

Hi,

I need the data to be in JSON format.

Netlify allows us to scale up and down, billing only for what is needed. I think that we'll be ok with Netlify for now after looking GCP pricing.

And I agree, let's get the MVP done first.

But I do need the test data because I need to assess memory usage and speed of processing.

Regards,

Bernard T. A. Baker

On Fri, Apr 17, 2020 at 7:55 AM un7c0rn notifications@github.com wrote:

@bernardbaker https://github.com/bernardbaker Sounds good can you provide a quick example of the format you need the test data to be in? I'm a little concerned about the Netlify quota. I'm assuming we can scale up requests for a fee as necessary? If we use GCP to hose the JS lambda functions we get 2M / month free: https://cloud.google.com/functions/pricing

That said, I'm also in favor of getting a working end-to-end testable MVP done and worrying about pricing optimizations as a second step.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/un7c0rn/stopthevirus/issues/46#issuecomment-615077340, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHNTTGZQG6M6C5NCSUT5UTRM74OXANCNFSM4MKC4GUA .