search

lingua-libre / operations

⚙️ Configuration files and deployment procedures for LinguaLibre wiki.
MIT License
0 stars 1 forks source link

Migrate `crontab` from lingualibre.fr to lingualibre.org ? #2

Open hugolpz opened 3 years ago

hugolpz commented 3 years ago

@Jitrixis, @Poslovitch : is one of you able to define what is ./crontab for ? Which skills are needed ?

I suspect it requires a Mediawiki / PHP / Backend expertises.

/crontab - understanding

/home/www/ actual folder structure and /crontab file's paths

Interpretation of results

Others

@mickeybarber, in the same server exploration you should bump into the following items which we would gain to document better...

Note: the 5 points above are nearly the same question = share with us the directory structure so we may see the possible broken paths. This will help us to know which url are outdated and by which to replace them.

Poslovitch commented 3 years ago

Crontab is short for "CRON Table", itself short for "Chrono table". It's basically a file that lists various cron scripts.

# Run maintenance scripts on the production instance
00 4 * * * /usr/bin/php7.0 /home/www/lingualibre.fr/maintenance/cleanupUploadStash.php > /dev/null 2>&1
00 5 * * * /usr/bin/php7.0 /home/www/lingualibre.fr/maintenance/rebuildLocalisationCache.php > /dev/null 2>&1

These scripts run maintenance php scripts everyday at 4 and 5 AM. Same goes for the testing instance, only at slightly different times.

/home/www/lingualibre.fr/maintenance/rebuildLocalisationCache.php, as an example, is the location of the php script that is run.

They also seem to send their logs to /dev/null. That either means these scripts tend to send a ton of logs, or that they just don't log anything. It might be worth trying to output their logs in a log file, to see what comes out.

# Other stuff
30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
45 2 * * 1 /bin/systemctl reload nginx
30 4 * * * logrotate /etc/logrotate.conf

These scripts reload nginx and eventually renew the SSL certificates with Let's Encrypt on Monday. And it daily does a logrotate. Never encountered that command so far, but based on the name, I guess it should get rid of older logs - that's a way to prevent them from clogging up huge amounts of disk space.

There's this interesting website that lets you "convert" the cron "timestamps" scripts : https://crontab.guru

Hope it helps.

hugolpz commented 3 years ago

@Poslovitch, thank you for this study an definition. Please consider to Github-Watch this repository so you can be the referent-mentor on this repository. :+1:

I think @mickeybarber (WM-fr with mission on WM-fr's servers) will check those servers paths to report on them when he can. With the info he will return we can decide what to do.

Poslovitch commented 3 years ago

Please consider to Github-Watch this repository so you can be the referent-mentor on this repository. 👍

@hugolpz Done.

What's interesting about this crontab is that it does not call the create_datasets.sh script. This might have been overlooked in the transition (.fr to .org), since datasets have stopped being updated around that time. Yet, the file's git history shows us that it was never there.

Should we add it in there?

Maybe it wasn't part of the crontab for a good reason (server load? disk space? backend/API issues?).

hugolpz commented 3 years ago

Wow. Good catch. From memory, 0x010C told me create_datasets.sh was run every odd day automatically. Need to find from where. (I'am off line for few days)

I suspect as soon as Michael will deploy the new create_datasets.sh with .org, datasets page will work again. But I don't know from where. If after a week it doesn't, then let add it in crontab ??

mickeybarber commented 3 years ago

Crons

projets.wikimedia.fr (webserver for mediawiki lingualibre.org):

lingualibre.wikimedia.fr (BlazeGraph and LL bot server)

Redirects

projets.wikimedia.fr (webserver for mediawiki lingualibre.org):

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name lingualibre.fr www.lingualibre.fr;

    ssl_certificate /etc/letsencrypt/live/lingualibre.fr/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/lingualibre.fr/privkey.pem;

    return 301 https://lingualibre.org$request_uri;
}

dev.lingualibre.fr => /home/www/dev.lingualibre.fr/ dev.lingualibre.org => /home/www/lingualibre.org/ v1.lingualibre.fr =>/home/www/v1.lingualibre.fr/ v2.lingualibre.fr => /home/www/v2.lingualibre.fr/ lingualibre.org -> /home/www/lingualibre.org/

Structures

lingualibre.wikimedia.fr (BlazeGraph and LL bot server)

hugolpz commented 3 years ago

Thanks a lot @mickeybarber ! I will digest your answer this weekend. :smiley: I will split this into several issue to focus on cronjobs here.