Migrate data from Filesystem to Postgres

[nelsonic]

At present the Node.js MVP saves data to the instance filesystem because that was the simplest way of storing data without having to manage any database. see: hits-nodejs/lib/db_filesystem.js#L26-L68

This worked well for MVP as it streams the contents of the file each time a request is made and counts the lines in the file as the count. Node.js shines at this because fs.createReadStream is non-blocking. Anyway, with the migration to phoenix, we need to:

• [ ] write an Elixir Script that parses the data in the filesystem and inserts it into PostgreSQL

Tasks

I estimate that this will take me T4h because I have to:

• [x] Log-in to the Linode instance and inspect how many directories have been created (this is the number of people - GitHub usernames - using the hits badge)
• [x] zip the data on Linode instance to get it off the instance - because I will be running the Elixir script on my localhost.
• [x] download the data to localhost
• [x] unzip data on localhost
• [x] Download and install Postgres on Mac: https://postgresapp.com/downloads.html
• [ ] write the script
  • [ ] insert data into hits_dev PostgreSQL on localhost
• [ ] run the script
• [ ] export the data from hits_dev PostgreSQL on localhost
• [ ] load the data on the remote server

[nelsonic]

While attempting to run the tar command:

tar -zcvf logs.tar.gz logs/

as per https://unix.stackexchange.com/questions/93139/can-i-zip-an-entire-folder-using-gzip we received the following error:

tar: logs: file changed as we read it

Now deleting the .gz files recursively:

find . -name "*.gz" -type f -delete

https://askubuntu.com/questions/377438/how-can-i-recursively-delete-all-files-of-a-specific-extension-in-the-current-di

Temporarily stopping the node.js server in order to perform the archive.

forever stop server.js

Now running:

tar -zcvf logs.tar.gz logs/

And it's taking forever ... ⏳

[nelsonic]

https://stackoverflow.com/questions/9427553/how-to-download-a-file-from-server-using-ssh Format:

scp your_username@remotehost.edu:foobar.txt /local/dir

Actual:

scp root@178.79.141.232:hits/logs.tar.gz ./logs.tar.gz

Worked:

[nelsonic]

Ecto refresher: https://geoffreylessel.com/2016/from-zero-to-ecto-in-10-minutes

[nelsonic]

Processing large files: https://www.poeticoding.com/processing-large-csv-files-with-elixir-streams

[nelsonic]

Roadblock: https://github.com/dwyl/learn-devops/issues/55

[nelsonic]

https://serverfault.com/questions/264595/can-scp-copy-directories-recursively

scp -rp root@178.79.141.232:hits/logs ./

scp -rp root@185.3.95.195:hits/logs ./

in-progress:

heading to bed let's see if it works overnight.

[nelsonic]

This is why we can't have nice things: