search

owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.39k stars 229 forks source link

Cannot get yarnAdminServer to run with mysql2 updates #3768

Closed toni-sharpe closed 4 months ago

toni-sharpe commented 4 months ago

This is fixed (possibly).

I was using startAdminServer and should have been using startAdminDevServer but I usually rely on my history in ITerm, not my memory, so must have accidentally had that in my history for some reason.

Still, all seems to be working for me with this so I'll close. Details belowe in case anyone wants a look, but there is still a problem with startAdminServer for me locally ...

I leave this for someone with better knowledge than me to close, details below.

================

## Description I ususally run the TMux alongside `yarn startAdminServer` so that I can get at the `:3030` port and develop locally. However, I think [this recent update](https://github.com/owid/owid-grapher/pull/3564) has caused problems there. It uses `mysql2` in Knex whereas from the error output below, it seems `yarn startAdminServer` requires `mysql` ``` ➜ owid-grapher git:(master) yarn startAdminServer node:internal/modules/cjs/loader:1078 throw err; ^ Error: Cannot find module 'mysql' Require stack: - /owid-grapher/itsJustJavascript/db/DatabaseConnection.js - /owid-grapher/itsJustJavascript/db/wpdb.js - /owid-grapher/itsJustJavascript/adminSiteServer/app.js at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15) at Module._load (node:internal/modules/cjs/loader:920:27) at Module.require (node:internal/modules/cjs/loader:1141:19) at require (node:internal/modules/cjs/helpers:110:18) at Object. (/owid-grapher/db/DatabaseConnection.ts:1:1) at Module._compile (node:internal/modules/cjs/loader:1254:14) at Module._extensions..js (node:internal/modules/cjs/loader:1308:10) at Module.load (node:internal/modules/cjs/loader:1117:32) at Module._load (node:internal/modules/cjs/loader:958:12) at Module.require (node:internal/modules/cjs/loader:1141:19) { code: 'MODULE_NOT_FOUND', requireStack: [ '/owid-grapher/itsJustJavascript/db/DatabaseConnection.js', '/owid-grapher/itsJustJavascript/db/wpdb.js', '/owid-grapher/itsJustJavascript/adminSiteServer/app.js' ] } Node.js v18.16.0 ``` ## Expected behaviour It should work with `mysql2` (or maybe, I am getting port `:3030` in an odd fashion and there is a better way, but even so, it's still broken). ## Steps to reproduce I think this shouold just happen if you pull down the latest master, run to install then try `yarn startAdminServer` ## Screenshots N/A see text output ## Environment MacOS M1 Sonoma 14.5 ITerm ## Notes Everything seems fine: the DB is visible in DBeaver, TMux is running without any error panels. Docker Desktop is working fine too. It's just this one command that won't start and let the `:3030` port work. My fork is synced, I am on the master branch and I have definitely pulled the code locally. ## Full outputs
Full TMux Docker out ``` Last login: Thu Jul 4 06:30:54 on ttys006 ➜ owid-grapher git:(master) make up ==> Checking your local environment has the necessary commands... ==> Validating your .env file for make up .env file valid for make up ==> Checking port ==> Building grapher yarn lerna run build lerna notice cli v8.1.3 ✔ @ourworldindata/types:build [existing outputs match the cache, left as is] ✔ @ourworldindata/utils:build [existing outputs match the cache, left as is] ✔ @ourworldindata/components:build [existing outputs match the cache, left as is] ✔ @ourworldindata/core-table:build [existing outputs match the cache, left as is] | | __, ,_ __, _ | | _ ,_ / | / | / | |/ \_|/ \ |/ / | \_/|/ |_/\_/|_/|__/ | |_/|__/ |_/ /| /| [+] Running 8/8 ✔ db 7 layers [⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 2.8s ✔ 022417830bcf Already exists 0.0s ✔ fe10b20dcdae Already exists 0.0s ✔ 624f2f22548b Already exists 0.0s ✔ 0eb07ea8ad2a Already exists 0.0s ✔ 8b517dfb74ad Already exists 0.0s ✔ ebef84d68e30 Already exists 0.0s ✔ 604975d7252e Already exists 0.0s [+] Building 0.1s (6/6) FINISHED docker:desktop-linux => [db-load-data internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 287B 0.0s => [db-load-data internal] load metadata for docker.io/mysql/mysql-server:latest 0.0s => [db-load-data internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [db-load-data 1/2] FROM docker.io/mysql/mysql-server:latest 0.0s => CACHED [db-load-data 2/2] RUN rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 && microdnf -y up 0.0s => [db-load-data] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:ed0a77ef9f76cf368844e714ae0d4855131ae47c0efbde8ef9c3f99141922fc9 0.0s => => naming to docker.io/library/owid-grapher-db-load-data 0.0s [+] Running 4/4 ✔ Network owid-grapher_default Created 0.1s ✔ Volume "owid-grapher_mysql_data_public" Created 0.0s ✔ Container owid-grapher-db-1 Created 0.1s ✔ Container owid-grapher-db-load-data-1 Created 0.1s Attaching to db-1, db-load-data-1 db-1 | [Entrypoint] MySQL Docker Image 8.0.32-1.2.11-server db-load-data-1 | [Entrypoint] MySQL Docker Image 8.0.32-1.2.11-server db-load-data-1 | Waiting for DB to be online db-1 | [Entrypoint] Initializing database db-1 | 2024-07-04T04:32:38.040473Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead. db-1 | 2024-07-04T04:32:38.040527Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead. db-1 | 2024-07-04T04:32:38.041084Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.32) initializing of server in progress as process 17 db-1 | 2024-07-04T04:32:38.062348Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. db-load-data-1 | mysqladmin: [Warning] Using a password on the command line interface can be insecure. db-1 | 2024-07-04T04:32:38.421445Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. db-1 | 2024-07-04T04:32:39.501331Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option. db-1 | [Entrypoint] Database initialized db-1 | 2024-07-04T04:32:43.649433Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead. db-1 | 2024-07-04T04:32:43.649908Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead. db-1 | 2024-07-04T04:32:43.649919Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32) starting as process 60 db-1 | 2024-07-04T04:32:43.660750Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. db-1 | 2024-07-04T04:32:43.726756Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. db-1 | 2024-07-04T04:32:43.816242Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. db-1 | 2024-07-04T04:32:43.816261Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. db-1 | 2024-07-04T04:32:43.824559Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.32' socket: '/var/lib/mysql/mysql.sock' port: 0 MySQL Community Server - GPL. db-1 | 2024-07-04T04:32:43.824906Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock db-1 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. db-1 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. db-1 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. db-1 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. db-1 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. db-1 | db-1 | [Entrypoint] ignoring /docker-entrypoint-initdb.d/* db-1 | db-1 | 2024-07-04T04:32:45.375296Z 11 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.32). db-1 | 2024-07-04T04:32:46.400173Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32) MySQL Community Server - GPL. db-1 | [Entrypoint] Server shut down db-1 | db-1 | [Entrypoint] MySQL init process done. Ready for start up. db-1 | db-1 | [Entrypoint] Starting MySQL 8.0.32-1.2.11-server db-1 | 2024-07-04T04:32:47.546567Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead. db-1 | 2024-07-04T04:32:47.546863Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead. db-1 | 2024-07-04T04:32:47.546875Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32) starting as process 1 db-1 | 2024-07-04T04:32:47.550801Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. db-1 | 2024-07-04T04:32:47.599880Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. db-1 | 2024-07-04T04:32:47.714196Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. db-1 | 2024-07-04T04:32:47.714216Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. db-1 | 2024-07-04T04:32:47.722064Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock db-1 | 2024-07-04T04:32:47.722096Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.32' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL. db-load-data-1 | Waiting for MySQL server to answer. db-load-data-1 | mysqld is alive db-load-data-1 | mysql: [Warning] Using a password on the command line interface can be insecure. db-load-data-1 | Checking if data dump has been downloaded db-load-data-1 | found owid_metadata.sql.gz db-load-data-1 | Creating user 'grapher' db-load-data-1 | mysql: [Warning] Using a password on the command line interface can be insecure. db-load-data-1 | ++ : grapher db-load-data-1 | ++ : grapher db-load-data-1 | ++ : grapher db-load-data-1 | ++ : db db-load-data-1 | ++ : 3306 db-load-data-1 | ++ : /tmp-downloads db-load-data-1 | ==> Refreshing grapher database db-load-data-1 | ++ fillGrapherDb db-load-data-1 | ++ echo '==> Refreshing grapher database' db-load-data-1 | ++ _mysql --database= -e 'DROP DATABASE IF EXISTS grapher;CREATE DATABASE grapher;ALTER DATABASE grapher CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs;' db-load-data-1 | ++ '[' -z grapher ']' db-load-data-1 | ++ mysql --default-character-set=utf8mb4 -hdb -ugrapher -pgrapher -P 3306 --database= -e 'DROP DATABASE IF EXISTS grapher;CREATE DATABASE grapher;ALTER DATABASE grapher CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs;' db-load-data-1 | mysql: [Warning] Using a password on the command line interface can be insecure. db-load-data-1 | Importing live Grapher metadata database (owid_metadata) db-load-data-1 | ++ '[' -f /tmp-downloads/owid_metadata.sql.gz ']' db-load-data-1 | ++ echo 'Importing live Grapher metadata database (owid_metadata)' db-load-data-1 | ++ import_db /tmp-downloads/owid_metadata.sql.gz db-load-data-1 | ++ cat /tmp-downloads/owid_metadata.sql.gz db-load-data-1 | ++ gunzip db-load-data-1 | ++ sed 's/.*DEFINER=`.*//' db-load-data-1 | ++ _mysql grapher db-load-data-1 | ++ '[' -z grapher ']' db-load-data-1 | ++ mysql --default-character-set=utf8mb4 -hdb -ugrapher -pgrapher -P 3306 grapher db-load-data-1 | mysql: [Warning] Using a password on the command line interface can be insecure. db-1 | 2024-07-04T04:33:18.682837Z 0 [Warning] [MY-013865] [InnoDB] Redo log writer is waiting for a new redo log file. Consider increasing innodb_redo_log_capacity. db-load-data-1 | ==> ✅ Grapher DB refresh complete db-load-data-1 | ✅ All done, grapher DB is loaded ✅ db-load-data-1 | ++ echo '==> ✅ Grapher DB refresh complete' db-load-data-1 | + return 0 db-load-data-1 | + echo '✅ All done, grapher DB is loaded ✅' db-load-data-1 exited with code 0 [grapher] 0:docker* 1:admin 2:vite 3:lerna 4:welcome- "Tonis-MacBook-Air.loc" 04:35 04-Jul-24 ```
Node modeules delete folder and with `--legacy-peer-deps` not good either ``` ➜ owid-grapher git:(master) rm -rf node_modules ➜ owid-grapher git:(master) npm install npm ERR! code ERESOLVE npm ERR! ERESOLVE unable to resolve dependency tree npm ERR! npm ERR! While resolving: grapher@3.0.0 npm ERR! Found: react@17.0.2 npm ERR! node_modules/react npm ERR! react@"^17.0.2" from the root project npm ERR! npm ERR! Could not resolve dependency: npm ERR! peer react@"^0.13.0 || ^0.14.0 || ^15.0.0 || ^16.0.0" from mobx-react@5.4.4 npm ERR! node_modules/mobx-react npm ERR! mobx-react@"5" from the root project npm ERR! npm ERR! Fix the upstream dependency conflict, or retry npm ERR! this command with --force or --legacy-peer-deps npm ERR! to accept an incorrect (and potentially broken) dependency resolution. npm ERR! npm ERR! npm ERR! For a full report see: npm ERR! /Users/brainstem/.npm/_logs/2024-07-04T04_40_05_653Z-eresolve-report.txt npm ERR! A complete log of this run can be found in: npm ERR! /Users/brainstem/.npm/_logs/2024-07-04T04_40_05_653Z-debug-0.log ➜ owid-grapher git:(master) npm install --legacy-peer-deps npm ERR! code EUNSUPPORTEDPROTOCOL npm ERR! Unsupported URL Type "workspace:": workspace:^ npm ERR! A complete log of this run can be found in: npm ERR! /Users/brainstem/.npm/_logs/2024-07-04T04_40_56_527Z-debug-0.log ➜ owid-grapher git:(master) ```
toni-sharpe commented 4 months ago

Hey @marcelgerber I see this is closed, could I know why? If it's a quick explanation, I'd appreciate the knowledge :-) Naturally, I understand if there's not time to explain.

marcelgerber commented 4 months ago

Ah, I was under the assumption that it's good to close now that you've fixed it.

As for fixing the yarn startAdminServer command itself, I'm pretty sure you can resolve that by running yarn cleanTsc && yarn lerna run build && yarn buildTsc.

Us in the team pretty much all use make up, which takes care of all of this automatically.

toni-sharpe commented 4 months ago

@marcelgerber fair enough, my immediate problem was missing the dev bit calling startAdminServer directly, which it sounds like I would never do, because of make up. Understood, thanks :-)