search

18F / 2015-foia-hub

A consolidated FOIA request hub.
Other
48 stars 17 forks source link

Unable to clone repo #776

Closed geramirez closed 9 years ago

geramirez commented 9 years ago

Data repo can't be cloned into the temp files to load agency contacts ERR Cloning into '/home/vcap/tmp/tmpq2pou6b2'...

khandelwal commented 9 years ago

So data loading was working before. Why is it broken now? We got the data in somehow the first time.

Sent from my iPhone

On May 1, 2015, at 4:51 PM, Gabriel Ramirez notifications@github.com wrote:

We can't clone the data repo into the temp files to load agency contacts ERR Cloning into '/home/vcap/tmp/tmpq2pou6b2'...

— Reply to this email directly or view it on GitHub https://github.com/18F/foia-hub/issues/776.

geramirez commented 9 years ago

So data loading was working before. Why is it broken now? We got the data in somehow the first time.

I'm asking about it but it looks like we are getting clone error across the board. 

2015-05-01T17:05:02.01-0400 [STG/0]      ERR Cloning into '/tmp/buildpacks/buildpack-python'...
2015-05-01T17:05:03.22-0400 [STG/0]      OUT Submodule 'compile-extensions' (https://github.com/cloudfoundry-incubator/compile-extensions.git) registered for path 'compile-extensions'
2015-05-01T17:05:03.24-0400 [STG/0]      ERR Cloning into 'compile-extensions'...
geramirez commented 9 years ago

Also, gunicorn is forcing app commands multiple times even though we only have one instance.

I downloaded the contact data into our repo and deployed again. 

2015-05-01T17:11:24.28-0400 [App/0]      OUT "----- Applying Migrations -----"
2015-05-01T17:11:24.31-0400 [App/0]      OUT "-----Loading Agency Contacts -----"
2015-05-01T17:11:24.35-0400 [App/0]      OUT "----- Applying Migrations -----"
2015-05-01T17:11:24.37-0400 [App/0]      OUT "-----Loading Agency Contacts -----"
2015-05-01T17:11:24.83-0400 [App/0]      OUT "----- Applying Migrations -----"
2015-05-01T17:11:24.83-0400 [App/0]      OUT "-----Loading Agency Contacts -----"
2015-05-01T17:11:24.90-0400 [App/0]      OUT "----- Applying Migrations -----"
2015-05-01T17:11:24.91-0400 [App/0]      OUT "-----Loading Agency Contacts -----"

The loading function appears to also be breaking because duplicates instances are loading data simultaneously.

khandelwal commented 9 years ago

Didn't we fix this with the instance number thing?

Sent from my iPhone

On May 1, 2015, at 5:24 PM, Gabriel Ramirez notifications@github.com wrote:

Also, gunicorn is forcing app commands multiple times even though we only have one instance.

I downloaded the contact data into our repo and deployed again.

2015-05-01T17:11:24.28-0400 [App/0] OUT "----- Applying Migrations -----" 2015-05-01T17:11:24.31-0400 [App/0] OUT "-----Loading Agency Contacts -----" 2015-05-01T17:11:24.35-0400 [App/0] OUT "----- Applying Migrations -----" 2015-05-01T17:11:24.37-0400 [App/0] OUT "-----Loading Agency Contacts -----" 2015-05-01T17:11:24.83-0400 [App/0] OUT "----- Applying Migrations -----" 2015-05-01T17:11:24.83-0400 [App/0] OUT "-----Loading Agency Contacts -----" 2015-05-01T17:11:24.90-0400 [App/0] OUT "----- Applying Migrations -----" 2015-05-01T17:11:24.91-0400 [App/0] OUT "-----Loading Agency Contacts -----"

The loading function appears to also be breaking because duplicates instances are loading data simultaneously.

— Reply to this email directly or view it on GitHub https://github.com/18F/foia-hub/issues/776#issuecomment-98241660.

geramirez commented 9 years ago

@khandelwal Yes, but this doesn't happen with waitress.

geramirez commented 9 years ago

I investigated the issue more closely today. git clone is working across the board even though it shows an error message. Also, Gunicorn appears to be running with the WEB_CONCURRENCY environment variable value set to 4 causing the app to start 4 times.

It seems that we'll have to design another way of controlling the loading data if we want to use Gunicorn and run more than 1 instance.

geramirez commented 9 years ago

This should fix it. 18F/foia-hub/pull/777

khandelwal commented 9 years ago

You are mentioning two different problems here. How are they related?

The first is that cloning is failing. The second is that loading data is being attempted multiple times.

Is the first happening also because of the second? How did you verify that?

geramirez commented 9 years ago

Initially I thought the cloning was failing because Cloud Foundry returns error for clonings like thisERR Cloning into '/home/vcap/tmp/tmpq2pou6b2'.... However, when I looked more closely I found that the cloning was actually working despite the error statements. The real reason the code was failing was because 4 instances of load_agency_contacts were running simultaneously which sometimes caused collisions.

ozzyjohnson commented 9 years ago

FYI - Late to the party, but just to confirm, those cloning errors are an effect of the way git sends certain messages to stderr. Nothing is actually failing.

For example:

git clone https://github.com/cloudfoundry/python-buildpack.git 2> errors.txt
cat errors.txt 
Cloning into 'python-buildpack'...

There's an open story about a future workaround for the confusion here:

https://www.pivotaltracker.com/n/projects/1042066/stories/85124412

@ramirezg @khandelwal