Currently, I have 2 workers for my little data pipeline
One is used to receive data from twitter and save to mongodb
One is using NLP to process the data.
NLP is the slow process for each item. Therefore, both 2 workers are set 100 concurrencies now but no matter how many concurrencies I set. The CPU is just used 1 core and 500Mb RAM only (I host it in GCP and requested 8 cores and 8Gb RAM for the vm)
Is there any way to optimize the CPU or RAM in order to boost up the processing power?
Currently, I have 2 workers for my little data pipeline One is used to receive data from twitter and save to mongodb One is using NLP to process the data.
NLP is the slow process for each item. Therefore, both 2 workers are set 100 concurrencies now but no matter how many concurrencies I set. The CPU is just used 1 core and 500Mb RAM only (I host it in GCP and requested 8 cores and 8Gb RAM for the vm)
Is there any way to optimize the CPU or RAM in order to boost up the processing power?