search

HiveGamesOSS / Chunker

Convert Minecraft worlds between Java Edition and Bedrock Edition
https://oss.chunker.app/
MIT License
272 stars 13 forks source link

Chunker not converting world and taking too long to download #147

Open kleines-L opened 3 hours ago

kleines-L commented 3 hours ago

Chunker Version

1.4.0-main-86becbf

Input World Version

Bedrock 1.21.40

Output World Version

Bedrock 1.21.40

Platform / Operating System

win32-x64-10.0.22631

Issue Description

Error Displayed: A fatal error occurred during conversion. Stack Trace: 

java.util.concurrent.CompletionException: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
    at com.hivemc.chunker.scheduling.task.Environment.cancel(Environment.java:85)
    at com.hivemc.chunker.conversion.WorldConverter.cancel(WorldConverter.java:609)
    at com.hivemc.chunker.conversion.WorldConverter.logFatalException(WorldConverter.java:441)
    at com.hivemc.chunker.scheduling.task.executor.TaskExecutor.handleException(TaskExecutor.java:109)
    at com.hivemc.chunker.scheduling.task.WrappedTask.lambda$setupFuture$0(WrappedTask.java:57)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
    at com.hivemc.chunker.scheduling.task.executor.TaskExecutorSupplier.run(TaskExecutorSupplier.java:44)
    at com.hivemc.chunker.scheduling.task.executor.TaskExecutor.threadLoop(TaskExecutor.java:178)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
    at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:550)
    at org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:1072)
    at com.hivemc.chunker.conversion.encoding.bedrock.v1_18_30.writer.ColumnWriter.writeMetadata(ColumnWriter.java:112)
    at com.hivemc.chunker.scheduling.function.ThrowableConsumer.invoke(ThrowableConsumer.java:20)
    at com.hivemc.chunker.scheduling.function.ThrowableConsumer.invoke(ThrowableConsumer.java:8)
    at com.hivemc.chunker.scheduling.task.WrappedTask.apply(WrappedTask.java:84)
    at com.hivemc.chunker.scheduling.task.WrappedTask$WrappedTaskConstant.apply(WrappedTask.java:129)
    at com.hivemc.chunker.scheduling.task.WrappedTask.get(WrappedTask.java:77)
    at com.hivemc.chunker.scheduling.task.executor.TaskExecutorSupplier.run(TaskExecutorSupplier.java:42)
    ... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.nio.HeapByteBuffer.<init>(Unknown Source)
    at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
    at org.iq80.leveldb.util.ZLib.uncompress(ZLib.java:49)
    at org.iq80.leveldb.table.Table.readRawBlock(Table.java:202)
    at org.iq80.leveldb.table.Table.readBlock(Table.java:157)
    at org.iq80.leveldb.table.Table.openBlock(Table.java:137)
    at org.iq80.leveldb.table.Table.openBlock(Table.java:121)
    at org.iq80.leveldb.table.Table.lambda$iterator$0(Table.java:116)
    at org.iq80.leveldb.table.Table$$Lambda$406/0x000000080022ddc0.apply(Unknown Source)
    at org.iq80.leveldb.iterator.TwoLevelIterator.initDataBlock(TwoLevelIterator.java:135)
    at org.iq80.leveldb.iterator.TwoLevelIterator.skipEmptyDataBlocksForward(TwoLevelIterator.java:108)
    at org.iq80.leveldb.iterator.TwoLevelIterator.internalNext(TwoLevelIterator.java:80)
    at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
    at org.iq80.leveldb.iterator.InternalTableIterator.next(InternalTableIterator.java:66)
    at org.iq80.leveldb.iterator.TwoLevelIterator.internalNext(TwoLevelIterator.java:80)
    at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
    at org.iq80.leveldb.iterator.MergingIterator.internalNext(MergingIterator.java:87)
    at org.iq80.leveldb.iterator.ASeekingIterator.next(ASeekingIterator.java:97)
    at org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1398)
    at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:638)
    at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:562)
    at org.iq80.leveldb.impl.DbImpl$$Lambda$424/0x000000080023ad18.run(Unknown Source)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 more

So, I tried converting a world and replacing glass with structure voids but every time I try it, the chunker app crashes at 40-50% converting status! If I try it on the website, everything works except for the download! It says it downloads the 130mb world in 50KB-500KB per second and after around 25% of the file being downloaded, the download speed goes to 0 B/s, then stops and cancels the download!

FormallyMyles commented 3 hours ago

For the running out of memory it may be related to https://github.com/HiveGamesOSS/Chunker/issues/125, or you don't have enough memory and might want to try a different PC.

Regarding the website, it might be related to your internet connection as worlds bigger than that usually download fine for me. You might want to try on a different web browser.

You can use the command-line version of the app and specify how much RAM you want to use if you're confident in doing that (the guide is in the README on the main GitHub page).

kleines-L commented 2 hours ago

I don't think that the problem is my internet connection but rather something server related to chunker. I tried it on a different browser and again the download speed is only 100kb/s to 500kb/s but I have a internet connection of 25MB/s! If I use the app it takes up to 5GB of RAM and a lot of CPU Power but at 50% it crashes randomly again! I don't really have another PC laying around better than mine. I have 16GB of ram and a intel i5-11600KF that should handle this. Usually it works but with this specific world not and I don't know why. It is mostly a void world with blocks placed from 0, 0 to 5k, 5k and a lot of structure voids. Could that maybe the problem?

Also can you guide me through using the command-line version because I can't really figure it out. Do I have to install something other than the app or the source code? And where do I get Java 17 from because I think that I have it already installed. And do you think it would be necessary to try this because chunker already uses enough RAM so I don't think that that should be the problem.

Screenshot 2024-11-24 132518 image image

FormallyMyles commented 2 hours ago

For big maps Chunker unfortunately can consume quite a bit of RAM (which seems like it given it's potentially 5000x5000).

Would you mind providing what input to output version / the goal of using Chunker is? (Since your original message said Bedrock 1.21.40 -> Bedrock 1.21.40).

The current algorithm for Chunker means the UI uses 25% of your RAM as a maximum, meaning it will only be using up to 4GB on your system, we'll hopefully improve this in the next release if you just want to wait for that (likely in the next few days).

In terms of running the command-line version, it would be opening command prompt in the directory of where you have the chunker-cli.jar and doing java -Xmx8G -jar chunker-cli.jar -i input_world_folder -o output_world_folder -f BEDROCK_R21_40. If you have Java installed it should in theory work, that command uses 8GB of ram as a maximum and you'd need to put the input world in the same folder / change the input_world_folder parameter. There are no other requirements other than Java 17 or above.

The current 1.4.0 chunker-cli jar is on our releases page - chunker-cli-1.4.0.jar.

Regarding the web version, it seems like the connection just gets interrupted, perhaps try something like Firefox / Edge assuming you're using Chrome. Otherwise, if you want to email the world to chunker-public@hivemc.com we can have the team look further into the connectivity issues to see if they can replicate it.