search

RoboBrainCode / Backend

Robobrain Backend - Django + uwsgi
http://robobrain.me:3000/
1 stars 0 forks source link

MongoDB space issue #9

Closed asheshjain399 closed 9 years ago

asheshjain399 commented 9 years ago

Mongodb is throwing "Can't take a write lock while out of disk space" on mongo-test. Though the disk usage is only 77%

arzavj commented 9 years ago

In fact, I don't think any feeds pushed to the api will be added to mongo as of now.

ozansener commented 9 years ago

It seems like it is actually out of disk space. Mongo allocates disk space in terms of 2GB blocks so if the remaining space is less than 2GB it fails. The problem is I even can not run anything since I can not get the write lock. So options are: 1- we scale the disk up but it might not work since we need to remap the disk as well 2-I can try to delete some not used database (the ones we experimented) but I can mess the entire system up. So, can you guys back-up all the data you actually need to mongo-prod then I will look at it.

arzavj commented 9 years ago

Could you remove unused tables since that is quick and if it works, that would be great? What are your concerns with that?

On Wed, Nov 19, 2014 at 4:26 PM, Ozan Sener notifications@github.com wrote:

It seems like it is actually out of disk space. Mongo allocates disk space in terms of 2GB blocks so if the remaining space is less than 2GB it fails. The problem is I even can not run anything since I can not get the write lock. So options are: 1- we scale the disk up but it might not work since we need to remap the disk as well 2-I can try to delete some not used database (the ones we experimented) but I can mess the entire system up. So, can you guys back-up all the data you actually need to mongo-prod then I will look at it.

— Reply to this email directly or view it on GitHub https://github.com/RoboBrainCode/Backend/issues/9#issuecomment-63740953.

ozansener commented 9 years ago

Basically, I can not run the dropTable command because of the space issue. For some reason that I do not understand, mongo requires disk space to drop tables as well. I assume it is basically failing to lock the DB due to the disk issue.

So I will force stop the database and delete the table manually then reload the other tables, I am just worried that sth can go wrong and I do not want to loose data.

asheshjain399 commented 9 years ago

I have cleaned some old mongo dump from the home directory. So the system is temporarily working.

But this is a temporary fix. We should increase the disk space. Only 110MB of disk memory is left on this machine.

On Wed, Nov 19, 2014 at 11:26 PM, Ozan Sener notifications@github.com wrote:

Basically, I can not run the dropTable command because of the space issue. For some reason that I do not understand, mongo requires disk space to drop tables as well. I assume it is basically failing to lock the DB due to the disk issue.

So I will force stop the database and delete the table manually then reload the other tables, I am just worried that sth can go wrong and I do not want to loose data.

— Reply to this email directly or view it on GitHub https://github.com/RoboBrainCode/Backend/issues/9#issuecomment-63770378.

Ashesh http://www.asheshjain.org

ozansener commented 9 years ago

The problem is apparently about aws configuration. Both mongo-test and mongo-prod actually has 200GB space but it is not mapped properly. I am looking at it.

ozansener commented 9 years ago

We mounted the missing drive and moved the mongodb to a new location with 200GB space. There is no space issue about mongo now.

asheshjain399 commented 9 years ago

Done other machines also have this issue, such as www-test? Can we map there disk to 200GB as well?

On Thu, Nov 20, 2014 at 1:37 AM, Ozan Sener notifications@github.com wrote:

We mounted the missing drive and moved the mongodb to a new location with 200GB space. There is no space issue about mongo now.

— Reply to this email directly or view it on GitHub https://github.com/RoboBrainCode/Backend/issues/9#issuecomment-63782585.

Ashesh http://www.asheshjain.org

adityajami commented 9 years ago

Looks like none of our instances never used the ebs until we discovered yesterday. Its as simple as doing a formatting of the volume using mkfs and modifying the fstab entries. May be we should include them in the getting started doc so that everyone remembers to do it when they spin up an instance.

Aditya

On Thu, Nov 20, 2014 at 9:57 AM, Ashesh Jain notifications@github.com wrote:

Done other machines also have this issue, such as www-test? Can we map there disk to 200GB as well?

On Thu, Nov 20, 2014 at 1:37 AM, Ozan Sener notifications@github.com wrote:

We mounted the missing drive and moved the mongodb to a new location with 200GB space. There is no space issue about mongo now.

— Reply to this email directly or view it on GitHub https://github.com/RoboBrainCode/Backend/issues/9#issuecomment-63782585.

Ashesh http://www.asheshjain.org

— Reply to this email directly or view it on GitHub https://github.com/RoboBrainCode/Backend/issues/9#issuecomment-63850638.