Suggestion: Easy On Boarding Flow & Documentation

[matikin9]

I'm going to try to run this within Cloud9.

I got this much to display:

This is the error I'm getting:

Executing (default): SELECT "id", "Geometry", "Fund_St", "Legacy_ID", "Lead_Ag", "Proj_Title", "Proj_Ty", "Proj_Desc", "Contact_info", "More_info", "Primary_Street", "Cross_Streets", "Proj_Status", "Proj_Man", "CD", "Access", "Dept_Proj_ID", "Other_ID", "Total_bgt", "Grant", "Other_funds", "Prop_c", "Measure_r", "Gas_Tax", "General_fund", "Authorization", "Issues", "Deobligation", "Explanation", "Constr_by", "Info_source", "Grant_Cat", "Grant_Cycle", "Est_Cost", "Fund_Rq", "Lc_match", "Flagged", "TotalUnmetFunding", "ProjectStartDate", "ProjectProjectedCompletionDate", "Dup_ID", "Attachment", "createdAt", "updatedAt" FROM "Projects" AS "Project" WHERE "Project"."Access" = 'Public';
Unhandled rejection SequelizeDatabaseError: column "TotalUnmetFunding" does not exist
    at Query.formatError (/home/ubuntu/workspace/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
    at null.<anonymous> (/home/ubuntu/workspace/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
    at emitOne (events.js:77:13)
    at emit (events.js:169:7)
    at Query.handleError (/home/ubuntu/workspace/node_modules/pg/lib/query.js:143:8)
    at null.<anonymous> (/home/ubuntu/workspace/node_modules/pg/lib/client.js:180:26)
    at emitOne (events.js:77:13)
    at emit (events.js:169:7)
    at Socket.<anonymous> (/home/ubuntu/workspace/node_modules/pg/lib/connection.js:136:12)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:153:18)
    at Socket.Readable.push (_stream_readable.js:111:10)
    at TCP.onread (net.js:537:20)
From previous event:
    at Query.run (/home/ubuntu/workspace/node_modules/sequelize/lib/dialects/postgres/query.js:74:17)
    at /home/ubuntu/workspace/node_modules/sequelize/lib/sequelize.js:849:20
    at /home/ubuntu/workspace/node_modules/sequelize/node_modules/retry-as-promised/index.js:40:21
From previous event:
    at retryAsPromised (/home/ubuntu/workspace/node_modules/sequelize/node_modules/retry-as-promised/index.js:30:10)
    at /home/ubuntu/workspace/node_modules/sequelize/lib/sequelize.js:848:12
From previous event:
    at Promise.then (/home/ubuntu/workspace/node_modules/sequelize/lib/promise.js:21:17)
    at Model.findAll (/home/ubuntu/workspace/node_modules/sequelize/lib/model.js:1395:6)
    at /home/ubuntu/workspace/controllers/projects_controller.js:56:22
    at Layer.handle [as handle_request] (/home/ubuntu/workspace/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/ubuntu/workspace/node_modules/express/lib/router/route.js:131:13)
    at Route.dispatch (/home/ubuntu/workspace/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/ubuntu/workspace/node_modules/express/lib/router/layer.js:95:5)
    at /home/ubuntu/workspace/node_modules/express/lib/router/index.js:277:22
    at Function.process_params (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:330:12)
    at next (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:271:10)
    at Function.handle (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:176:3)
    at router (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:46:12)
    at Layer.handle [as handle_request] (/home/ubuntu/workspace/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:312:13)
    at /home/ubuntu/workspace/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:330:12)
    at next (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:271:10)
    at /home/ubuntu/workspace/node_modules/express/lib/router/index.js:618:15
    at next (/home/ubuntu/workspace/node_modules/express/lib/router/index.js:256:14)

[matikin9]

I tracked down the error - the db migration file doesn't contain 3 fields from the Project model: TotalUnmetFunding, ProjectStartDate, and ProjectProjectedCompletionDate. Thus, columns don't exist for them in the database and I get an error when trying to start the app and query the db.

[crisr10]

I think I can help you with that. You just need to delete the database, run the app, so the new columns get added to a not yet existing database and then do the migration one more time. I was part of the team who worked on the latest version of the app :)

[matikin9]

Hm, I'm not sure how that would work... if the database doesn't exist when I run npm start it throws a SequelizeConnectionError.

Shouldn't the Project table's migration script define TotalUnmetFunding, ProjectStartDate, and ProjectProjectedCompletionDate? Those fields are in the model.

[crisr10]

You get an error because there are 3 fields in the model that are not in the migration (The ones mentioned above). If you delete the database, and then start the app, Sequelize will create a new database with all the fields in the model (Including TotalUnmetFunding, ProjectStartDate, and ProjectProjectedCompletionDate). Now that the fields are added, then you do the migration on this new database with all the fields and will have no more issues. What you can do next, is add these changes to the migration.

Just wanted to share some insight since we all had to do this at some point this at some point, but forgot to add this to the migration file

[crisr10]

As long as you have the right login information in the config file (to connect to your local PostgreSQL) when you start the app, it creates the database

[matikin9]

Interesting... I thought I did delete the database (keeping the same connection info as before) but when I ran the app, it threw an error saying there's no database to connect to. I'll have to do some more research into this to figure out what's going on. BTW, thanks for taking the time to respond!

Would it break anything if I just add the fields to the migration file? I'm trying to document an easy on-boarding process for people attending Civic Hack Night.

[hunterowens]

@matikin9 I do not think that will be an issue.

[crisr10]

I want to make a correction. The database does need to exist for Sequelize to connect. That's why you need to specify the database on the config file. What you can do is drop all the existing tables and start the app. Then add the fields to the existing migration file

[matikin9]

@crisr10 Sweet, I've made progress. The issue I'm running into now is when I try to run node load.js. The issue seems to be that in /models/index.js, the environment variable NODE_ENV is not defined in process.env.NODE_ENV, and thus the config variable is undefined.

I thought that having these lines in package.json would have set the environment variable:

"scripts": {
    "start": "NODE_ENV=development node ./bin/www"
},

[matikin9]

Alrighty - I added instructions for setting the environment variable within Cloud9. Now my error is:

$ ./transform_data.sh 
psql: FATAL:  database "dot" does not exist

Am I supposed to do anything to the ATD_project database.gdb.zip file, which is sitting within \etl\data?

[matikin9]

I think this is the issue I'm seeing currently. Inside the transform_data.sh file, the below line is trying to open a postgresql database named dot:

# remove temp tample 
psql -d dot -c "DROP TABLE ogrgeojson"

However, based on following the instructions in the README file so far, these are the only databases I have:

postgres=# \l
                              List of databases
   Name    |  Owner   | Encoding | Collate |  Ctype  |   Access privileges   
-----------+----------+----------+---------+---------+-----------------------
 postgres  | postgres | UTF8     | C.UTF-8 | C.UTF-8 | 
 template0 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
           |          |          |         |         | postgres=CTc/postgres
 template1 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
           |          |          |         |         | postgres=CTc/postgres
(3 rows)

Was dot supposed to be created locally at some point in the installation process?

[hunterowens]

@matikin9 the transform_data.sh script is only for the "real" data, which isn't open due to privacy concerns.

The data script you want to run and follow is for the db/projects.js file, which has fixtures data on how to load the app. Once you load the fixtures data and create an account (projecturl/users/signup) I think you should be good to go.

[matikin9]

@hunterowens I see - so we don't need to do the Loading Data portion of the README? Perfect :ok_hand: we're good to go then:

[hunterowens]

👍

[hunterowens]

@matikin9 can you review readme changes in last commit and close issue?

[matikin9]

Looks good! I've updated the README on my fork to break down the instructions for using Cloud9: https://github.com/matikin9/dot-planner