search

Odonno / surrealdb-migrations

An awesome SurrealDB migration tool, with a user-friendly CLI and a versatile Rust library that enables seamless integration into any project.
https://crates.io/crates/surrealdb-migrations
MIT License
210 stars 16 forks source link

Large migration does not terminate #43

Closed unidesigner closed 11 months ago

unidesigner commented 1 year ago

I am trying to run a "large" migration (about 130k update statements) and the output displays "Executing migration..." for >10min but I don't see any updates/changes in the SurrealDB log. Splitting the migration into two migrations with 60k statements runs in <1min. What could be the issue here?

Odonno commented 1 year ago

Hi Stephan,

This is really interesting. I suppose that running long migration can take time but indeed > 10min is very long. I have no idea what could the cause for now.

Will you be able to provide a reproducible example project?

Thank you.

unidesigner commented 1 year ago

Hi David,

I give you access to the repo where I'm working on the migration. I created a new branch which shows the problem (long-migration for migration AddSegments_FlyWire). The migrations in the main branch work quickly (where I split the problematic migration in two).

Would be great if you could debug this!

Odonno commented 1 year ago

Thank you Stephan,

Your project in your main branch indeed takes almost 1 minute to complete which is already feels really long to me. I will try to see why it is taking so long.

Odonno commented 1 year ago

Here is the bottleneck: https://github.com/Odonno/surrealdb-migrations/blob/f5f35dc1652133f6d741db3bf4bdbb34a197f6e5/src/surrealdb.rs#L106

I suppose we need to do more checks on surrealdb part.

unidesigner commented 1 year ago

Ah, I did not think the issue is with the main branch, but making it go even faster would be nice. The issue I'm seeing is in the long-migration branch which does not terminate.

I don't have the skills to debug the original surrealdb rust code, unfortunately. I hope you do. Otherwise, we could try and ask on the SurrealDB Discord.

Odonno commented 1 year ago

What I can reproduce is that the more UPDATE statements you have in a transaction, the longer it takes. I mean each statement after the other is longer. I do not know why at the moment.

It will take some time for me to debug, you can try the discord channel or even create an issue on the GitHub repo of surrealdb.

unidesigner commented 1 year ago

FYI. I brought this issue up at the developer office hours and it's on the radar. https://github.com/surrealdb/surrealdb/issues/1810 https://github.com/surrealdb/surrealdb/issues/1227 but not high priority at the moment. "our highest priority at the moment is backward compatibility and we're working hard to address this in the Beta 10 release this month. After that, we will be looking into this performance issue, but not sure what the status is at the moment"

Odonno commented 11 months ago

Hi @unidesigner

I have released a new version of surrealdb-migrations that now targets 1.0.0. Does that improve performance in any way? As it is a surrealdb issue, I suppose we can close this issue?

Anyway, you can download the latest release of surrealdb-migrations: 

cargo install surrealdb-migrations@1.0.0-preview.1