hydron import takes too much memory

[Chiiruno]

Perhaps have hydron check how much memory there is, allocate in chunks of 256MiB, until there is X amount of memory total left unused for other programs? I'd rather my entire computer not lock, please.

[bakape]

Working on it.

On 25 July 2018 at 03:09, チルノ notifications@github.com wrote:

Perhaps have hydron check how much memory there is, allocate in chunks of 256MiB, until there is X amount of memory total left unused for other programs? I'd rather my entire computer not lock, please.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsDQ_FL0pw0Mp7BVjD1V4icCouJlDks5uJ7dPgaJpZM4VfO19 .

[bakape]

Also, see https://github.com/bakape/thumbnailer/issues/24

On 25 July 2018 at 03:13, meguca meguca scorpid33@gmail.com wrote:

Working on it.

On 25 July 2018 at 03:09, チルノ notifications@github.com wrote:

Perhaps have hydron check how much memory there is, allocate in chunks of 256MiB, until there is X amount of memory total left unused for other programs? I'd rather my entire computer not lock, please.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsDQ_FL0pw0Mp7BVjD1V4icCouJlDks5uJ7dPgaJpZM4VfO19 .

[Chiiruno]

The Go FFI would create this much memory bloat on import?

[bakape]

Not as much, but it's not negligible. Also, I think the crux of your problem is scanning file you already have imported. That creates tons of garbage.

On 25 July 2018 at 03:15, チルノ notifications@github.com wrote:

The Go FFI would create this much memory bloat on import?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-407592291, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsL_VrdzqA9GhZjc_QeTvhSgjz0Puks5uJ7izgaJpZM4VfO19 .

[Chiiruno]

I remove the .hydron folder, drop the db, recreate the db, have hydron initialize normally, add db_conf.json, then have hydron initialize with postgres.

[bakape]

Is this still relevant?

[Chiiruno]

I'll have to check later, but even without that, I noticed something you did, not sure what, did reduce memory usage earlier. Only about 8GiB was being used, which is still a lot, but much better than everything+16GiB swap. I'll test and close if it's significantly better in a bit.

[bakape]

The Go runtime will naturally increase its memory pool, if there is more available, and won't return it to the OS immediately.

On 13 August 2018 at 14:25, チルノ notifications@github.com wrote:

I'll have to check later, but even without that, I noticed something you did, not sure what, did reduce memory usage earlier. Only about 8GiB was being used, which is still a lot, but much better than everything+16GiB swap. I'll test and close if it's significantly better in a bit.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-412487234, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsKl0nb1CD3xDJyvCBOdED7r9_nuiks5uQWIjgaJpZM4VfO19 .

[Chiiruno]

Are we reusing memory where possible? Maybe GODEBUG=gctrace=1 might help here?

[Chiiruno]

Or, if it's because of the goroutines, maybe clean them up more often at the expense of some speed. I feel garbage is piling up with the import goroutines and we're probably not cleaning it often enough, thus why the Go runtime keeps increasing it's memory pool. That might have changed with your recent commit, though.

[bakape]

The Go runtime will always increase heap memory, if it can. Welcome to GC languages.

On 13 Aug 2018 2:56 pm, "チルノ" notifications@github.com wrote:

Or, if it's because of the goroutines, maybe clean them up more often at the expense of some speed. I feel garbage is piling up with the import goroutines and we're probably not cleaning it often enough, thus why the Go runtime keeps increasing it's memory pool. That might have changed with your recent commit, though.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-412493965, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsAsbh7DYMLNnOebFj6QWViuHlWWbks5uQWmBgaJpZM4VfO19 .

[Chiiruno]

aaaaaaaaaaaaaaaaaaaaaa this is why i hate garbage collection

[bakape]

Rewrite thumbnailer in Rust and we could honestly rewrite hydron to it.

On Mon, 13 Aug 2018, 15:22 チルノ, notifications@github.com wrote:

aaaaaaaaaaaaaaaaaaaaaa this is why i hate garbage collection

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-412499882, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsCh2CCNeU_sgR0uEte8OABkmt2O5ks5uQW-JgaJpZM4VfO19 .

[Chiiruno]

Rust isn't ready. I don't want to write shit in it until 2.0, where it won't be shit. 1.0 was a fucking farce. Could rewrite it in C++, but at that point might as well wait for Rust 2.0.

[Chiiruno]

Also, at least in Rust or C/C++, we can keep tabs on how much memory there is available, and allocate where necessary in chunks. I think that might be possible in Go, but I feel it may be counter-productive with the GC.

[bakape]

Or provide a pure C/C++ interface for the existing thumbnailer. That is probably the least painful route and still accomplish our goals.

On Mon, 13 Aug 2018, 15:25 meguca meguca, scorpid33@gmail.com wrote:

Rewrite thumbnailer in Rust and we could honestly rewrite hydron to it.

On Mon, 13 Aug 2018, 15:22 チルノ, notifications@github.com wrote:

aaaaaaaaaaaaaaaaaaaaaa this is why i hate garbage collection

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-412499882, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsCh2CCNeU_sgR0uEte8OABkmt2O5ks5uQW-JgaJpZM4VfO19 .

[Chiiruno]

That could work, but at the same time the thumbnailer is small enough that you might as well rewrite it in C++ too. You're right, though. C/C++ interface until Rust 2.0 is probably the most productive path.

[bakape]

The brunt of it is already in C so you really just need to rewrite the little Go logic there to C and still keep a Go interface to it.

On Mon, 13 Aug 2018, 15:31 チルノ, notifications@github.com wrote:

That could work, but at the same time the thumbnailer is small enough that you might as well rewrite it in C++ too. You're right, though. C/C++ interface until Rust 2.0 is probably the most productive path.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bakape/hydron/issues/32#issuecomment-412501983, or mute the thread https://github.com/notifications/unsubscribe-auth/AHfPsPpYOT4BZCvFxMLSslBLUlioZY62ks5uQXGjgaJpZM4VfO19 .

[Chiiruno]

Importing already imported files memory usage is SIGNIFICANTLY better, I think it only used like 200MiB total, if even that. Good job, that's like night and day. The initial import seems slightly better memory-wise, so I think we need to focus on that now. It's your call if you want to close this issue now or later, but the initial import is still pretty big, and would be really bad for people with 100,000+ images.

[Chiiruno]

Also I think it skips fetching tags too if already imported, so that means my nix expression won't have to DDoS Gelbooru.

[Chiiruno]

Also, since #23 is a problem, and the new import is good, do you think we should close #23 ?