search

DocNow / hydrator

Turn Tweet IDs into Twitter JSON & CSV from your desktop!
MIT License
434 stars 64 forks source link

Limit of records #58

Closed Nabilah95 closed 4 years ago

Nabilah95 commented 4 years ago

Hi, I want to know the limitation of records that Hydrator can hydrate. Will I be able to hydrate a total amount of 125,680,841 Twitter API?

edsu commented 4 years ago

I just hydrated 32 million ok. It took about a week but i didn't run it continuously. I did notice it periodically stopped hydrating. I relinked the account and then things started working again. For 125 million you might want to use twarc on a server, that doesn't turn off. But hydrator should work too.

Nabilah95 commented 4 years ago

I just hydrated 32 million ok. It took about a week but i didn't run it continuously. I did notice it periodically stopped hydrating. I relinked the account and then things started working again. For 125 million you might want to use twarc on a server, that doesn't turn off. But hydrator should work too.

Thank you for responding, may I know what is the RAM of your laptop? Is it 4, 8 or 16 GB RAM?

edsu commented 4 years ago

8GB,  but RAM shouldn't be an issue. It is mostly network bound. Definitely use the latest version of the Hydrator (v0.0.13) since it fixes a CPU performance issue that was in some of the earlier versions.You may want to try a smaller dataset to see how it goes before committing to hydrating such a large dataset. You will also want to make sure you have enough available disk space.