search

FinalsClub / karmaworld

KarmaNotes.org v3.0
GNU Affero General Public License v3.0
7 stars 6 forks source link

migrate TheFinalsClub and other stuffs #370

Closed btbonval closed 9 years ago

btbonval commented 10 years ago

This ticket is not entirely appropriate to this project, but unfortunately we lack a good place for this ticket. Since moving to Heroku primarily concerns KarmaNotes, the ticket goes here.

btbonval commented 10 years ago

Beta Karmanotes on EC2 is shutdown, but not terminated.

btbonval commented 10 years ago

TheFinalClub database has been converted to PostgreSQL, but needs to be uploaded into a Heroku database. TheFinalClub code was pushed to Heroku without issue, although it doesn't run yet without the database.

AndrewMagliozzi commented 10 years ago

Sweet

On May 24, 2014, at 1:34 PM, Bryan Bonvallet notifications@github.com wrote:

TheFinalClub database has been converted to PostgreSQL, but needs to be uploaded into a Heroku database. TheFinalClub code was pushed to Heroku without issue, although it doesn't run yet without the database.

— Reply to this email directly or view it on GitHub.

btbonval commented 10 years ago

ugh I forgot in PHP world there is no database wrapper library. All the code is specifically written for MySQL. So that conversion was possibly a waste and I have to figure out how to get MySQL in this piece. For Heroku, MySQL only supports 5 MB at the free tier, and this database is at least 41 MB.

btbonval commented 10 years ago

for Node.js FinalsClub (circa 2011)

updated bc submodule to point at an existing github project (was pointing at finlsclubdev github user which no longer exists).

found the correct Mongo database on the same system as LectureLeaks.org. dumped it and restored it into a MongoLab Heroku-connected database. Should be free, database was 300some MiB and MongoLab supports nearly 500 MiB for free.

rewrote the mongo uri in app.js because it assumed the database name would be fc, but we do not get to name the database with MongoLab via Heroku.

Wrote one line Procfile: web: node app.js

It's running at http://vast-taiga-6101.herokuapp.com/ but has an application error.

2014-05-25T05:09:14.929624+00:00 heroku[web.1]: Starting process with command `node app.js`
2014-05-25T05:09:16.115442+00:00 app[web.1]: 
2014-05-25T05:09:16.115954+00:00 app[web.1]: /app/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:10
2014-05-25T05:09:16.116837+00:00 app[web.1]:                                                                         ^
2014-05-25T05:09:16.116172+00:00 app[web.1]: number') throw "host and port must be specified [" + host + ":"  + port + "]";
2014-05-25T05:09:16.109879+00:00 app[web.1]: No hostname defined, defaulting to os.hostname(): eb1618c5-701a-41eb-9e48-03a3d8bfe9e4
2014-05-25T05:09:16.116842+00:00 app[web.1]: host and port must be specified [ds033489.mongolab.com:33489]
2014-05-25T05:09:11.984811+00:00 heroku[web.1]: Process exited with status 8
2014-05-25T05:09:17.316906+00:00 heroku[web.1]: State changed from starting to crashed
2014-05-25T05:09:17.301792+00:00 heroku[web.1]: Process exited with status 8

I found that error on Stackoverflow, but it was not helpful. http://stackoverflow.com/questions/18777274/cant-get-the-mongo-portion-of-my-heroku-app-to-work

Looks like a problem of int vs string with internal variables? https://groups.google.com/forum/#!topic/derbyjs/fxmVak-HOCY

Found a connect-mongo library which had this internal variable problem back in 2011, fixed in 2012 with a version change to 0.1.7. The connect-mongo version specified in package.json is 0.1.6. https://github.com/kcbanner/connect-mongo/issues/15

Updated and pushed to Heroku. Now it can't bind to the listening port.

2014-05-26T01:37:45.253594+00:00 app[web.1]: Express server listening on port 3000 in development mode
2014-05-26T01:37:45.236188+00:00 app[web.1]:    info  - socket.io started
2014-05-26T01:37:44.125913+00:00 heroku[web.1]: Starting process with command `node app.js`
2014-05-26T01:38:09.533546+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=GET path=/ host=vast-taiga-6101.herokuapp.com request_id=793c72b7-6bdd-4e71-969a-2338d0151bc7 fwd="76.109.230.235" dyno= connect= service= status=503 bytes=
2014-05-26T01:38:44.703883+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2014-05-26T01:38:44.704129+00:00 heroku[web.1]: Stopping process with SIGKILL
2014-05-26T01:38:46.734144+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path=/ host=vast-taiga-6101.herokuapp.com request_id=ee5c4ae6-0906-4da3-bb46-f57540e6fb45 fwd="76.109.230.235" dyno= connect= service= status=503 bytes=
2014-05-26T01:38:46.055238+00:00 heroku[web.1]: State changed from starting to crashed

Need to set SERVER_HOST and SERVER_PORT env vars. SERVER_HOST set to the Heroku domain. SERVER_PORT set to 80 fails to bind, so it must be the internal port, which is exposed from Heroku as PORT. Updating code to match. var serverPort = process.env.PORT || process.env.SERVER_PORT

That got it running for one visit. The styling looked awful. Console logs threw up all over player.css (it's bad), and player.js errored. Future attempts got 500 internal server error. Appears to be related to mongo.

2014-05-26T01:47:43.034665+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=vast-taiga-6101.herokuapp.com request_id=c94a4bdb-58ed-470a-9fc4-d8e5550c1f10 fwd="76.109.230.235" dyno=web.1 connect=1ms service=7ms status=500 bytes=203
2014-05-26T01:47:43.028040+00:00 app[web.1]: MongoError
2014-05-26T01:47:43.028056+00:00 app[web.1]:     at commandHandler (/app/node_modules/mongodb/lib/mongodb/cursor.js:472:21)
2014-05-26T01:47:43.028053+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/mongodb/cursor.js:472:39
2014-05-26T01:47:43.028057+00:00 app[web.1]:     at g (events.js:180:16)
2014-05-26T01:47:43.028046+00:00 app[web.1]:     at Db.wrap (/app/node_modules/mongodb/lib/mongodb/db.js:1044:11)
2014-05-26T01:47:43.028048+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/mongodb/collection.js:903:74
2014-05-26T01:47:43.028050+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/mongodb/cursor.js:130:30
2014-05-26T01:47:43.028051+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/mongodb/cursor.js:174:32
2014-05-26T01:47:43.028054+00:00 app[web.1]:     at Cursor.close (/app/node_modules/mongodb/lib/mongodb/cursor.js:688:5)
2014-05-26T01:47:43.028059+00:00 app[web.1]:     at EventEmitter.emit (events.js:106:17)
2014-05-26T01:47:43.028061+00:00 app[web.1]:     at Db._callHandler (/app/node_modules/mongodb/lib/mongodb/db.js:756:25)

According to MongoLab, they are running mongod version 2.4.9. package.json calls for mongodb module version 0.9.7-2-5 (newest version is 1.3.20). Unclear how the versions relate to each other, but mongodb module version 1.2.14 fixed support for node 0.10, which is the currently running. The README for FinalsClub says node.js version 0.4.10. Added that to packages.json under engines/node, but same error occurs. Not sure if Heroku is actually using the specified node version.

Updated mongodb for >0.9.7-2-5 (so newest version). Seemed to work. Page loaded, CSS is good, everything looks right.

Compared against old.finalsclub.org. It's very different. Turns out there was a bunch of work done in the workspace and in an unlabeled branch on the Linode system by Seth and Jordan. Emailed Seth to find out more about that unlabeled branch.

btbonval commented 10 years ago

Found an email Bob sent a long time ago about how to turn on LectureLeaks. The instructions were wrong (he said run python manage.py in /var/www/LectureLeaks, but there's no Django in there). It dawned on me to search for "manage.py" under that directory, and I found where it has been hiding the whole time. I can now attempt to migrate that to Heroku.

Code is uploaded to github.com/FinalsClub/LectureLeaks.git, but it was apparently out of date with what is on the hard drive. Pushed newer code (mostly mode changes).

Pushed media (copied out of Django admin code, since it was a soft link), static, and 5 GiB uploads to S3. Rewrote Django config to hopefully use static and media from S3, but unsure how to point "uploads" to a static directory elsewhere in Django.

dj_django database dumped from Linode, pushed to S3, and loaded onto Heroku with heroku pgbackups:restore 'https://lectureleaks-backup.s3.amazonaws.com/LL.pg.dump'. No errors. I see 295 courses in the documents_course table, so that probably worked.

Need to rewrite the config files for ENV VARS where possible and hit the go button.

AndrewMagliozzi commented 10 years ago

Sweet

On May 26, 2014, at 12:12 PM, Bryan Bonvallet notifications@github.com wrote:

Found an email Bob sent a long time ago about how to turn on LectureLeaks. The instructions were wrong (he said run python manage.py in /var/www/LectureLeaks, but there's no Django in there). It dawned on me to search for "manage.py" under that directory, and I found where it has been hiding the whole time. I can now attempt to migrate that to Heroku.

— Reply to this email directly or view it on GitHub.

btbonval commented 9 years ago

From an email dated May 19, 2014 regarding thefinalclub.org

It looks like certain file uploads were sent to disk. I was afraid the PHP wiki system built in would store wiki data to file (which is common in PHP land), but it only stored images to disk.

Running junk on Heroku means having a transient disk store. Here's a fellow who wanted to get around the transient disk store: http://stackoverflow.com/questions/15214118/local-disk-space-on-heroku-for-storing-a-lucene-index-for-read-only

Basically, you don't.

Something we might be able to do is run GridFS over MongoDB to simulate a file system, which can be run on Heroku, and then we'd have to rewrite chunks of this PHP code to utilize the GridFS somehow.

btbonval commented 9 years ago

This seems to be as migrated as we're going to go. The Linode will persist for now.