Open ThatOneCalculator opened 1 year ago
Thanks for sharing the instructions, very handy.
That is something we should implement fully on our side to automate it. But that remains to be done. This isn't easy. And to be honest it's not a priority for me, it's already quite demanding to package new releases.
To do it manually, from what I can understand after a quick look, it seems the first part of the script that applies patches can be done in Yunohost without any issue. The Calckey install part is a bit more tricky.
My naive try would be:
This is a classical method for software that share the same kind of database (it will import it), assuming they are at least a bit compatible (= it can start and run migrations if needed).
To be tested in a non critical situation (you may loose everything, except if your backup is good), it will surely fail somewhere.
That seems like a reasonable approach. I will try it and tell you what happens.
Okay. Well, I tried that. I copied the "db.sql" file and the "files" directory in my YunoHost Misskey backup into my YunoHost Calckey backup (that is, replaced the Calckey versions) and then restored.
Intriguingly, the files directory from Calckey was not empty -- examining it, I find content from my contacts. I assume it must be getting federation messages based on the domain being the same as before?
It comes up with "Internal Server Error" in my browser, even after several reloads.
I restarted the "Calckey" service a couple of times to see if that would help. The log of (the 3rd?) restart looked like this:
Jun 11 05:13:07 systemd[1]: Stopping Calckey: fork of Misskey...
Jun 11 05:13:07 npm[3879024]: Shutting down: received [SIGTERM] signal
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] federation: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] notes: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] users: Write skipped
Jun 11 05:13:07 npm[3879010]: Shutting down: received [SIGTERM] signal
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] activeUsers: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] instance: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] perUserNotes: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] drive: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] perUserReaction: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] hashtag: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] perUserFollowing: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] perUserDrive: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [chart] apRequest: Write skipped
Jun 11 05:13:07 npm[3879024]: INFO 1 [core] The process is going to exit with code 0
Jun 11 05:13:07 npm[3879010]: INFO [chart] federation: Write skipped
Jun 11 05:13:07 npm[3879010]: INFO [chart] notes: Write skipped
Jun 11 05:13:07 npm[3879010]: INFO * [chart] users: Write skipped
Jun 11 05:13:07 npm[3878874]: undefined
Jun 11 05:13:07 npm[3878874]: /var/www/calckey/packages/backend:
Jun 11 05:13:07 npm[3878874]: ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL backend@ start: pnpm node ./built/index.js
Jun 11 05:13:07 npm[3878874]: Command failed with signal "SIGTERM"
Jun 11 05:13:07 systemd[1]: calckey.service: Succeeded.
Jun 11 05:13:07 systemd[1]: Stopped Calckey: fork of Misskey.
Jun 11 05:13:07 systemd[1]: calckey.service: Consumed 21.500s CPU time.
Jun 11 05:13:07 systemd[1]: Started Calckey: fork of Misskey.
Jun 11 05:13:08 npm[3880158]: > calckey@13.1.4.1 start
Jun 11 05:13:08 npm[3880158]: > pnpm --filter backend run start
Now, I'm wondering if that "pnpm" command will eventually migrate the database or something? So I'm not sure if this is actually a failure, yet, or just that I haven't given it enough time. The database backup was about 980 MB and there's about 20GB of data in files. So it might take awhile to finish processing them if it needs to do that. (?)
It's still running, but it produces a lot of "missing column" errors, like:
Jun 11 06:44:33 npm[2689]: WARN 1 [queue inbox] failed(QueryFailedError: column Meta.disableRecommendedTimeline does not exist) id=283 attempts=4/8 age=11m activity=https://social.vivaldi.net/users/mykillenmy#delete
Here is the limit of what I can help to debug, I have no idea where to start because of my lack of *key internal DB structure knowledge.
Okay. Well, I thought there might be some auto-healing built in to recover from the missing columns issues, but it hasn't recovered after running overnight, so I'm guessing not.
Seems like this method fails. I will have to take some time to decide whether it's worth pursuing a more complex solution, or just write off the loss of either my Misskey instance/accounts or the Calckey software. I'm not sure how attached I really am to either, so either or both are certainly options.
It has been an interesting experiment. Thank you very much for your help!
Seems like this method fails.
I'm not sure I wasn't clear, but I think this method might work, but with adjustments because I don't know which step to run at which point, in particular for the migration and patch stuff. But I'm not able to tell nor test which adjustments need to be done, sorry.
Yeah. I understood. I'm currently restoring the Misskey installation. If a solution is suggested by someone on this thread, while I'm still using Misskey, I'll likely give it a try again.
It's still running, but it produces a lot of "missing column" errors, like:
Jun 11 06:44:33 npm[2689]: WARN 1 [queue inbox] failed(QueryFailedError: column Meta.disableRecommendedTimeline does not exist) id=283 attempts=4/8 age=11m activity=social.vivaldi.net/users/mykillenmy#delete
That means that migrations weren't run. Is there any way you can manually run pnpm run migrate
in the Calckey directory?
Okay. I have repeated the process as before. Except, I then installed pnpm:
# npm install -g pnpm
Moved to the Calckey directory:
# cd /var/www/calckey
And ran the proposed command:
# pnpm run migrate
It is not a happy camper, though. I'll paste the entire output here. Sorry for the length, but I'm not sure which bit is important! It looks like maybe there is something I have to clean up first?
root@ynh:/var/www/calckey# pnpm run migrate
> calckey@13.1.4.1 migrate /var/www/calckey
> pnpm --filter backend run migrate
packages/backend | WARN The field "resolutions" was found in /var/www/calckey/packages/backend/package.json. This will not take effect. You should configure "resolutions" at the root of the workspace instead.
> backend@ migrate /var/www/calckey/packages/backend
> typeorm migration:run -d ormconfig.js
query: SELECT * FROM current_schema()
query: SELECT version();
query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'public' AND "table_name" = 'migrations'
query: CREATE TABLE "migrations" ("id" SERIAL NOT NULL, "timestamp" bigint NOT NULL, "name" character varying NOT NULL, CONSTRAINT "PK_8c82d7f526340ab734260ea46be" PRIMARY KEY ("id"))
query failed: CREATE TABLE "migrations" ("id" SERIAL NOT NULL, "timestamp" bigint NOT NULL, "name" character varying NOT NULL, CONSTRAINT "PK_8c82d7f526340ab734260ea46be" PRIMARY KEY ("id"))
error: error: relation "migrations" already exists
at Parser.parseErrorMessage (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
length: 104,
severity: 'ERROR',
code: '42P07',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'heap.c',
line: '1160',
routine: 'heap_create_with_catalog'
}
Error during migration run:
QueryFailedError: relation "migrations" already exists
at PostgresQueryRunner.query (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async PostgresQueryRunner.executeQueries (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/query-runner/BaseQueryRunner.js:421:13)
at async PostgresQueryRunner.createTable (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:410:9)
at async MigrationExecutor.createMigrationsTableIfNotExist (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/migration/MigrationExecutor.js:345:13)
at async MigrationExecutor.executePendingMigrations (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/migration/MigrationExecutor.js:129:9)
at async DataSource.runMigrations (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/data-source/DataSource.js:249:35)
at async Object.handler (/var/www/calckey/node_modules/.pnpm/typeorm@0.3.11_zfbzyadn2rnu3hf3pysto5crhm/node_modules/typeorm/commands/MigrationRunCommand.js:68:13) {
query: 'CREATE TABLE "migrations" ("id" SERIAL NOT NULL, "timestamp" bigint NOT NULL, "name" character varying NOT NULL, CONSTRAINT "PK_8c82d7f526340ab734260ea46be" PRIMARY KEY ("id"))',
parameters: undefined,
driverError: error: relation "migrations" already exists
at Parser.parseErrorMessage (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/var/www/calckey/node_modules/.pnpm/pg-protocol@1.5.0/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
length: 104,
severity: 'ERROR',
code: '42P07',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'heap.c',
line: '1160',
routine: 'heap_create_with_catalog'
},
length: 104,
severity: 'ERROR',
code: '42P07',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'heap.c',
line: '1160',
routine: 'heap_create_with_catalog'
}
/var/www/calckey/packages/backend:
ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL backend@ migrate: `typeorm migration:run -d ormconfig.js`
Exit status 1
ELIFECYCLE Command failed with exit code 1.
root@ynh:/var/www/calckey#
Could you edit it to use "`" instead ? It would be a lot more readable
Thank you for documenting this.
I mean, this traceback is unfamiliar to me, but it looks to me like the relevant bit is probably:
QueryFailedError: relation "migrations" already exists
Just to be clear about my skills: Quite skilled with Linux and Debian (which YunoHost is based on). I can probably install or run software if needed. Moderately skilled with YunoHost, mostly through the web, though I have used its CLI interface a few times.
Kind of on the edge with PostgreSQL. I know RDBMS conceptually and I've used MySQL. I think I have the Adminer YunoHost app installed, so I probably could use that to look at the database, but I have no experience with it, yet.
Totally out of my depth with npm, pnpm, Typsecript, modern Javascript libraries, Node.js (when I did Javascript many years ago, it was a little toy language you could use for stupid HTML stricks).
I use Python for most of my coding needs, so it's a whole other environment. Once upon a time I wrote C, C++, and Fortran, but it's been decades.
We will definitely need someone with some understanding of that stuff to investigate… For my part I'm hopeless.
Can I ask what version of Misskey you're migrating from?
Misskey version: 12.119.2~ynh1
Calckey version: 13.1.4.1~ynh1
YunoHost version: 11.1.20
Postgresql version: 13.11-0+deb11u1
Also, after some fiddling, I found that the backup archive also contains the database name, username, and password -- all for "calckey". These would be the ones from the newly-created Calckey that I backed up to create the merged archive.
But I can get into that with Adminer using these credentials, if it would be useful to get some table info? (But I don't know what to look for).
Or, if it's simpler, I can probably do the same with psql on the command line.
I'm assuming that the db.sql file just loads the database contents, so they will be in this 'calckey' database now, even though they came from the 'misskey' database before.
I tried using the misskey credentials and database, but Adminer gives me an authentication error. I suspect that database no longer exists after uninstalling misskey, but I'm unsure how else to prove that.
I'm assuming that the db.sql file just loads the database contents, so they will be in this 'calckey' database now, even though they came from the 'misskey' database before.
If you restored a calckey backup, yes it does.
I suspect that database no longer exists after uninstalling misskey, but I'm unsure how else to prove that.
Yes. Uninstall an app removes everything, including the database (ex: https://github.com/YunoHost-Apps/misskey_ynh/blob/master/scripts/remove#L60).
Okay, one thing that looks a little off: I got into psql using the credentials for calckey (so I'm user 'calckey' in database 'calckey'), and I list the tables in the database:
calckey=> \dt
The list shows all the tables are owned by postgres
which I gather is like a superuser:
List of relations
Schema | Name | Type | Owner
--------+---------------------------------+-------+----------
public | __chart__active_users | table | postgres
public | __chart__ap_request | table | postgres
public | __chart__drive | table | postgres
public | __chart__federation | table | postgres
public | __chart__hashtag | table | postgres
public | __chart__instance | table | postgres
[...]
I was thinking that maybe I could just trying deleting the migrations
table and then run
pnpm run migrate
But it won't let me do that as calckey
because the table is owned by postgres
.
Is that right? Seems like the point of having a Calckey database and username would be to interact as calckey
.
I'm not 100% sure I can log in as postgres
, but I'll try it, then I should be able to change the ownership to calckey
, right..?
(I still have the merged backup, of course, so if I wreck it, I can start over).
The list shows all the tables are owned by postgres which I gather is like a superuser:
I need to check the owner on a normal installation, but that probably has to be corrected 🤔. Yunohost should create it under Calckey user. It might be because it was Misskey user, and has it doesn't exit anymore, it felt back to postgres…
I'm not 100% sure I can log in as postgres, but I'll try it, then I should be able to change the ownership to calckey, right..?
Yes, I don't remember where they are but you can have the credentials/log as root or something.
Rather than try to fix the live database (which I think may exceed my Postgresql skills), I am editing the 'db.sql' backup file.
It seems it is NOT safe to do a global substitution of misskey
to calckey
, because it appears in things like remote URLs for federated sites, etc. In fact, there are almost a million references to 'misskey' in this file.
But there are many OWNER TO misskey
cases that can be subbed to OWNER TO calckey
, and I also went ahead and substituted Owner: misskey
, which seems to only be in comments (but there's a LOT more of those, >1000, which is slightly worrying, compared to the 138 cases of OWNER TO misskey
).
I then attempted to restore this. I got the missing column errors in the log. I stopped the calckey service.
I ran 'pnpm run migrate' and this time, it did not report errors (it did give me a long list of queries, ending with "COMMIT", which seems like a successful result. I saved the output if you want to see it). The only potentially suspicious line I noticed was:
packages/backend | WARN The field "resolutions" was found in /var/www/calckey/packages/backend/package.json. This will not take effect. You should configure "resolutions" at the root of the workspace instead.
So then I restarted calckey, and I think I'm making progress, because I can access the application (screenshot), though it reports "An error has occurred".
Visiting my settings shows me my profile. Initially, my banner image showed correctly, but my avatar was missing (apparently Calckeys shows colorbars for this). But after a few minutes, visiting my settings showed colorbars for my banner as well.
In the log, I have a lot of errors like this:
Jun 14 22:12:39 npm[2588412]: Bad system call
Jun 14 22:12:41 npm[2588412]: Bad system call
Jun 14 22:12:42 npm[2588425]: WARN 1 [queue inbox] failed(EntityNotFoundError: Could not find any entity of type "User" matching: {
Jun 14 22:12:42 npm[2588425]: "id": "8zygxcz2x7"
Jun 14 22:12:42 npm[2588425]: }) id=37 attempts=3/8 age=4m activity=https://retro.social/users/ajroach42/statuses/110544845389885471/activity
Jun 14 22:12:43 npm[2588412]: Bad system call
Jun 14 22:12:45 npm[2588412]: Bad system call
The "id" hash this is showing looks a lot like fields I see in __chart__per_user_drive
in the db.sql file. So the fact that data in my user drive is missing might be related?
Trying to visit the timeline just gives me an error bird, like the screenshot below.
This db.sql file is awkwardly large, so I tried creating a fresh Misskey site, backing that up, and examining that (MUCH smaller) db.sql file. However this didn't turn up anything interesting. Other than the OWNER TO misskey
and Owner: misskey
references, the only others are some Github URLs and a "featured page" for "about-misskey".
So, I don't know what remains to be done, but the site is still broken. Less broken, which is encouraging, but still broken.
I saved the output if you want to see it
Yes please, if it contains no personal information, that might be valuable later, at least to reproduce it and check if it's the same :)
So then I restarted calckey, and I think I'm making progress, because I can access the application (screenshot), though it reports "An error has occurred".
🎉 well done !
Can you check the status of calckey service (service calckey status
) ?
So, I don't know what remains to be done, but the site is still broken. Less broken, which is encouraging, but still broken.
Thanks a lot for your efforts in trying to make this work, we're making progress :)
Here's output from the status query:
root@ynh:~# service calckey status
● calckey.service - Calckey: fork of Misskey
Loaded: loaded (/etc/systemd/system/calckey.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-06-15 08:35:38 UTC; 6min ago
Main PID: 2924724 (npm start)
Tasks: 57 (limit: 4696)
Memory: 549.7M
CPU: 6min 30.135s
CGroup: /system.slice/calckey.service
├─2924724 npm start
├─2924943 sh -c pnpm --filter backend run start
├─2924944 node /opt/node_n/n/versions/node/19/bin/pnpm --filter backend run start
├─2924970 sh -c pnpm node ./built/index.js
├─2924971 node /opt/node_n/n/versions/node/19/bin/pnpm node ./built/index.js
├─2924982 Calckey (master)
└─2924997 Calckey (worker)
Jun 15 08:41:46 ynh.lunaticsproject.org npm[2924997]: "id": "900dl1eyyc"
Jun 15 08:41:46 ynh.lunaticsproject.org npm[2924997]: }) id=594 attempts=3/8 age=4m activity=https://mastodon.art/users/AnnaOutOfSp4ce/status>
Jun 15 08:41:48 ynh.lunaticsproject.org npm[2924982]: Bad system call
Jun 15 08:41:50 ynh.lunaticsproject.org npm[2924982]: Bad system call
Jun 15 08:41:52 ynh.lunaticsproject.org npm[2924982]: Bad system call
Jun 15 08:41:54 ynh.lunaticsproject.org npm[2924982]: Bad system call
Jun 15 08:41:56 ynh.lunaticsproject.org npm[2924982]: Bad system call
Jun 15 08:41:57 ynh.lunaticsproject.org npm[2924997]: WARN 1 [queue inbox] failed(EntityNotFoundError: Could not find any entit>
Jun 15 08:41:57 ynh.lunaticsproject.org npm[2924997]: "id": "99tdq579kr"
Jun 15 08:41:57 ynh.lunaticsproject.org npm[2924997]: }) id=609 attempts=3/8 age=4m activity=https://newsie.social/users/ChrisBoese/statuses/>
lines 1-26/26 (END)
Note the EntityNotFoundError
-- the log has a LOT of these, interspersed among the vague Bad system call
messages. It seems to be trying to look up information about a remote user? The id
is obviously some kind of hash. But I really don't know what's going on there. But ChrisBoese
is someone I do follow.
Are these possibly "federation errors"?
And I'm sorry, I spoke too soon about the output from pnpm run migrate
. I had pasted it into Kate, but I hadn't saved yet and Kate crashed (probably due to using it on db.sql). I'd have to repeat the procedure to get to that again (probably going to happen, though. I'll post it when I get there again).
Ok, we might need to full one (I don't remember the command) but that might be quite normal log I think, I have the same in my instance… In my case I understand it as federation errors yes.
This may be a matter of degree. I get the impression that the Calckey instance can't find ANY of its resources. It reports a stream of users it can't find, and once I deleted cookies for the site from my browser, it goes back to trying to create a admin user. This fails if I try to use any of my old accounts, and I see this in the log:
Jun 15 17:46:29 npm[3129540]: ERR 1 [api] Internal error occurred in admin/accounts/create: USED_USERNAME
So I try creating a new admin user, and I am able to log in. But no posts or notifications show up anywhere (interestingly, it is saying "No posts" now, rather than "An error occurred"). But as I am logged in as admin, I can now get to the control panel / dashboard:
I poked around to find out what I might be able to see from the control panel, but the results are not encouraging:
It doesn't list ANY users, except the newly-created admin. It doesn't list ANY files (there are 17 GB of files in /var/www/calckey/files). It shows no instances federating. It does have my site description. It does have my server block list. It does list my relay. The job queue is empty/idle. It lists all the "custom emoji" names -- but NONE of the images (all colorbars).
If I use the "Lookup" to look up user accounts and enter local accounts I know existed on the previous site, it reports not found. If I search for a remote account I know exists, also "User not found". If I put the fully qualified Fediverse ID for a local account @LunaticsTV@m.filmfreedom.net
it does not report anything (doesn't pop-up the "User not found" message, but still does not show anything).
Basically, aside from a few form fields, it doesn't seem to have gotten anything useful from the database -- it's like a new site with no content, it seems?
If I go to the users panel, I can see the details for the new admin account (the only user it lists). If I hit the +
icon and try to create a user with my old username TerryHancock
it pops up an unspecific error message "Internal error occurred".
If I try to create a TerryHancock2
it makes a new user.
So it seems that it is colliding with the old data, but not actually able to use it.
Interestingly, I do see that it tracked -4 users and -237 posts, which means that at the very least, chart data is preserved.
So it is working to some degree 😅
Yes ?
Currently we have instructions on how to migrate standard deployments of Misskey to Calckey here: https://codeberg.org/calckey/calckey/src/branch/develop/docs/migrate.md
However, someone recently asked how to do that for YunoHost, and sadly I don't have many pointers to give. If @oufmilo or someone else could elaborate on that process I'd be happy to add it to the official docs.