Reproduce development website on exmachinis.com production server

[darozak]

Fernando, Can you reproduce the advolition.com website on exmachinis.com?

[freelancefrivas]

Ok, I'm uploading the files now, which will take a while. Should I override the database too? And do you want to push everything to the master branch in the repository?

[darozak]

Fernando, I've added Carlos (@crodnum) to this issue so he can provide input on the best way to manage this migration.

• I think the exmachinis.com DB should be overwritten except for the user and drone tables. They should remain unchanged. However, we should backup the data before overwriting just to be safe.
• Carlos, any reasons why we shouldn't push dev to master branch in repository (to accompany the migration)? Makes sense to me.
• Carlos and Fernando, can we create a list (if it doesn't already exist) of all the directories that need to be moved from dev to prod servers during these migrations? Including best way of dealing with DB migration?

I'm planning to be more diligent about keeping the exmachinis.com server current in the future so it would help to have a procedure for doing so.

Thanks! Hope all is well.

Dave

[freelancefrivas]

The folders to be moved on my side would be inside the /var/www folder: app, bootstrap, config, database, html, node_modules, resources, routes, storage, tests and vendor. No need to move the files in the root folder. I've been thinking about a script to merge databases but I can't come up with a solution. Maybe Carlos has something to say about this. I would usually solve this by using Laravel migrations, but it would be unfair asking you to learn that system that works on PHP. But in case you do agree to use that system (creating a PHP file for each new change in the 'database/migrations' folder), I'd recommend creating a GIT repository in Exmachinis and one in Advolition and create hooks so the migrations run after each new push.

[crodnun]

Hi guys,

No problem merging develop into master branch. Develop branch has the latest features added.

I will prepare a quick document (asap) with the steps required to do these steps. I will need to refresh the status of the exmachinis server and compare it with advolition.

Regarding DB update, I will also prepare a quick guide with the latest changes we added and how I would do it. I think that we should do it manually this time but no problem using migrations in the future. I often do this with https://flywaydb.org/, probably is similar to your method. Isn't it Fernando?

The advantage of flyway is that you do not need to program the migrations just configure the SQL steps to be done. Maybe it is easier for everybody and we avoid using PHP coding, I do not mind.

[freelancefrivas]

Ok, @crodnun, I didn't know about that, I'll investigate it. I was proposing we use one repository in Exmachinis and one in Advolition, but I just remembered we work in different folders... At the moment I can't make up my mind. Maybe Each one will have to do their own scripts and maybe even repositories to be able to update code/db in one server and then push the changes to the other. Were you thinking in using Flyway together with GIT?

[freelancefrivas]

I've been checking Flyway a bit now. It seems cool. As I understand you create a new file for each new version that has SQL code inside. Maybe I'm over complicating things with my proposal, and for now we can just add a folder for these migrations, share it through GIT, and run the migrations locally each time we pull, and remotely each time we push... The thing is, right now every time I make changes in the project I have to: push to the repository (dev or master), upload the files through FTP, run the migration files on the server, and also re-build the javascript package and update composer (if there were changes related). That could all be done automatically if I put one repository in each server and make a script, but at the moment we can leave that for later...

[crodnun]

Hi guys, I have shared here a quick document with the steps that are required to upgrade current DB and game engine at ex-machinis server.

You have also here have a reference to the folders involved at game engine side.

Hope it helps, @darozak if you want I help you with this just let me know it. I wrote these documents just to let you play around and get used to the basic chores required there, feel free to drop me questions if something does not work as intended.

[darozak]

Thanks Carlos and Fernando for your thoughts and input! I agree that it's best to do this manually, at least at first. And it helps me to learn my way around the system! The instructions that Carlos put together are particularly helpful. I'll see if I can follow them without any trouble :) Fernando, I also appreciate your list of directories that need to be transferred to support the website! I give this a shot. I suspect problems will arise and when they do I'll be in touch!

[freelancefrivas]

So you'll move the database then? Have in mind the users and agents tables have different structures in both schemas, if you want to merge them...

[darozak]

Ok. I updated the database and the game engine. The game engine appears to be running correctly based on the log. However, it doesn't appear to be processing any emails that I send it. I'm not even getting a response from emails sent to registrar@exmachinis.com. Nor is the website online. I'm guessing Laravel needs to but updated for the email server and website?

Fernando (@freelancefrivas), can you take a look?

[freelancefrivas]

Hello. Apparently some PHP module wasn't working with the PHP version installed (7.0), so I upgraded to 7.4. I think it works well now, but the mail handling is not being automatically called. Just in case, @crodnun, I updated packages on the exmachinis server... (called apt-get update).

[darozak]

Thanks Fernando! The website appears to be working perfectly with the exception of the companies page: https://exmachinis.com/companies. Can you see if you can fix it?

[freelancefrivas]

Ah, yes, there was an error that wasn't reported in PHP 7.2 but it came up in 7.4 now, so I fixed it.

[darozak]

Awesome! The companies page is displaying perfectly. However, the game-engine doesn't look like it's receiving email commands again. Nor is the registrar responding. The game-engine does appear to be running correctly (based on the log). I'm not sure what happened because it had been processing emails properly after your PHP upgrade. Fernando, can you take a look?

[freelancefrivas]

As far as I remember, it's the game engine that calls the PHP script on each loop, right? In the past I had used a cron job to do that. When I call the script it seems to work fine, it's just that is not being called. And by the way: I forgot, was it OK to push the latest changes to the master?

[darozak]

Thanks Fernando. What's the specific PHP script that needs to be called?

[darozak]

I think I found the problem. It appears that the mail is still being received by a crontab call to the mail handler script. However, the mail handler script is hard coded for advolition.com. I haven't had a chance to fix it yet but it shouldn't be hard.

[freelancefrivas]

I found the url "hard-coded" at start_email_handler.sh and changed it to exmachinis.com (instead of advolition.com), but I guess that would need re-compiling and I'm not sure about how to do it. On the other hand, I see the call in the crontab, but it's commented out. Do you want me to activate it there?

[darozak]

Yes. Could you try turning on the call in the crontab.

[freelancefrivas]

Ok, it should be triggered every 10 minutes now...

[darozak]

Thanks Fernando! What is the path the file that you updated?

[freelancefrivas]

No problem! I'm not sure what you refer to. The cron tab is edited using crontab -e. The file where I found the url was start_email_handler.sh, in the 'scripts' folder. I have to say I found a crontab.txt containing these lines: */1 * * * * sh -c '. $HOME/.profile; $PLAT_HOME/bin/scripts/email_handler.sh' >> /dev/null 2>&1 */1 * * * * sh -c '. $HOME/.profile; $PLAT_HOME/bin/scripts/start_game_engine.sh' >> /dev/null 2>&1 So maybe those should be called from using cron jobs? Let's hope @crodnun can return soon and let us know...

[crodnun]

Hi guys, I had the chance to review the exmachinis server today (apologies for the delay), and the only weird thing that I noticed was the following, some swap files locked there:

 8 -rw-r--r-- 1 forth dev   5412 Apr 11 17:04 eforth.blk.old
  4 -rwxr--r-- 1 forth dev   1076 Apr 11 17:04 housekeeping.sh
 52 -rwxr-xr-x 1 forth dev  52928 Apr 11 17:04 events-engine.17092019
 32 -rwxr-xr-x 1 forth dev  29696 Apr 11 17:04 events-engine.09082019
  4 -rwxr-x--- 1 forth dev   1082 Apr 11 17:04 stop_game_engine.sh
  8 -rw-r--r-- 1 forth dev   5422 Apr 11 17:04 embed_mod2.blk
116 -rwxr-xr-x 1 forth dev 118776 Apr 11 17:04 game-engine.03232020
 68 -rwxr-xr-x 1 forth dev  69528 Apr 11 17:04 game-engine.07122018
 60 -rwxr-xr-x 1 forth dev  60080 Apr 11 17:04 game-engine.11092018
 76 -rwxr-xr-x 1 forth dev  74760 Apr 11 17:04 game-engine.05052019
 84 -rwxr-xr-x 1 forth dev  83552 Apr 11 17:04 game-engine.07062109
  4 -rwxr-x--- 1 forth dev   1115 Apr 11 17:04 stop_email_handler.sh
 88 -rwxr-xr-x 1 forth dev  86264 Apr 11 17:04 embed
  8 -rw-r--r-- 1 forth dev   5412 Apr 11 17:04 embed_mod1.blk
  4 -rwxr-x--- 1 forth dev   1102 Apr 11 17:04 stop_events_engine.sh
  4 drwxr-xr-x 3 forth dev   4096 Apr 11 17:04 ..
  4 -rwxr-x--- 1 forth dev   3003 Apr 16 00:17 start_email_handler.sh
  4 -rw-r--r-- 1 forth dev   1024 Apr 16 11:49 ..env.swp
  **4 -rw-r--r-- 1 forth dev   1024 Apr 18 00:16 .housekeeping.sh.swp**
  **4 -rw-r--r-- 1 forth dev   1024 Apr 18 00:18 .start_email_handler.sh.swp**
  4 drwxr-xr-x 3 forth dev   4096 Apr 18 00:18 .

The rest was correctly configured so I did the following steps as forth user:

1 - Firstly, stop the whole system: stop_system.sh 2 - Remove these swap files 3 - Start the system again: start_system.sh

Testing from my email everything seems working now, I am able to send and receive emails from this server and of course, execute commands, receive their output, etc,...

@darozak , feel free to jump in now and test it out.

Cheers Carlos

[crodnun]

Another minor detail detected at config file, probably just a consequence of using advolition config to update this server:

At file /home/forth/game-engine/conf/game-engine.conf we had this parameter incorrectly configured:

AGENTS_EMAIL_DOMAIN = "advolition.com"

replaced by:

AGENTS_EMAIL_DOMAIN = "exmachinis.com"

This was just a minor detail, basically, the response emails had the wrong domain but this is also fixed now.

[crodnun]

Another minor change, again the same problem just using advolition.com domain.

To answer emails, we have under /home/forth/game-engine/bin/scripts/email several scripts whose only purpose is to send emails responses generated by the game engine, there we also need to configure the right domain to receive the right domain.

Minor change again, also fixed.

I have updated all these steps in our guide for migrations under doc folder, this way we all know them for the future.

[darozak]

Thanks Carlos! The email is working nicely. I'll know what to look for next time :)

[darozak]

Hi @crodnun . I was able to fix a couple errors in the way the resources were being bought and sold on the advolition server. Everything is working great there!

However, after copying the updated game-engine directory over to the exmachinis server, I couldn't get the exmachinis server back online. As before, the game-engine and events-engine appear to be working fine (generating proper log files). However, the system isn't responding to emails.

I checked for locked swap files but didn't see any. I also updated the config and email settings for the exachinis.com domain.

Sorry to bother you with this again. But, could you take a look and see if you can figure this out?

[crodnun]

Hi @darozak ,

I have reviewed the same and only detected this issue:

[forth@ExMachinis] # pwd
/home/forth/game-engine/bin/scripts

  4 -rwxr-x--- 1 forth dev   3003 Apr 27 07:57 start_email_handler.sh.issue#101
  4 -rwxr-x--- 1 forth dev   3003 Apr 27 07:57 start_email_handler.sh

You had there (at file with extension .issue#101 the advolition.com URL instead of exmachinis.com).

[forth@ExMachinis] # cat start_email_handler.sh.issue#101 | grep URL
# URL where we check emails presence
EMAIL_URL=https://www.advolition.com/get-mails/JJy3CC9cUtzsbLsY

Anyways, after fixing it and restarting the system, seems that we keep having issues when reading emails. My guess is that the key configured there is not up-to-date or something is wrong with SMTP and email configuration. This was configured by Fernando and not sure if it has changed recently.

If you paste this on your browser:

https://www.exmachinis.com/get-mails/JJy3CC9cUtzsbLsY

We should be able to retrieve emails sent but we are not able to retrieve them, so my guess is that the key is not correct.

@freelancefrivas , could you help us with this, I am not able to detect what is happening there, is the key correct? why this URL is not able to retrieve emails?

[crodnun]

My advice, to update servers is to just copy the following binaries when you work at development side:

/home/forth/game-engine/bin/game-engine
/home/forth/game-engine/bin/events-engine

The rest of pieces never change once configured unless we decided to add new configuration.

Working at C codes, only updating these 2 binaries is enough to patch exmachinis.com server

[freelancefrivas]

Sorry for the delay. It's strange, it seems the configuration file (.env) got overwritten with the Advolition one. I don't remember doing that. In any case, it was calling the Advolition mail server, it should work now.

[darozak]

Awesome. The email handler is back online. Thanks to both of you for helping with this! I'm going to adopt Carlos' (@crodnun) suggestion of only updating those files that have changed. However, I'm also compiling a list of all the files that do have server specific parameters, which we'll need to keep an eye on.

Fernando ( @freelancefrivas ), where is the .env file located?

[freelancefrivas]

It's in the root folder. That file shouldn't be overwritten. I think the best procedure would be this: Copy everything except the .env file, the node_modules folder and the vendor folder from one side to the other (the last two are taken out just because they are heavy). Then, from the console in the root folder, run composer update (in case PHP packages were added) and npm update (in case there were Javascript changes), and then npm run prod to re-build the app.js.

[darozak]

Thanks Fernando! I'll give that a try.