Grid observer is unusably slow

[TGlas]

Adding the following grid observer:

<ObservationFromGrid>
<Grid name="Blocks">
<min x="-10" y="-10" z="-10"/>
<max x="10" y="10" z="10"/>
</Grid>
</ObservationFromGrid>

(21 x 21 x 21 = 9261 blocks per observation) to my mission slows down the game (or the Malmö mod?) so much that the system is unusable. I see two possible behaviors, and I did not figure out yet when I get which of the two:

• The controller keeps sending commands, but the game freezes for up to several seconds, then moves on for very few frames, then freezes again.
• The controller keeps spitting out commands, the game keeps moving, but the new commands are ignored and the game keeps repeating the very first commands (key press state).

Reporting 10K blocks should not be a big deal, even at 20Hz, compared to rendering the scene. So I suspect that there is probably a source of inefficiency somewhere. The problem is "scalable" in the sense that requesting fewer blocks results in shorter lags, but also for 1K blocks the lags are clearly percievable.

[timhutton]

Hi Tobias,

It's certainly something we can improve, if there's a need, but we're a bit surprised that you want a grid observation of that size. What are you trying to achieve?

Tim

[TGlas]

Dear Tim, I am interested in working on "high level" tasks involving hierarchical and/or long-term planning. I am not very keen on working on 3D vision, be it with machine learning or CV engineering approaches. I would happily throw a "3D-from-video" (SLAM-style) algorithm at the vision input to extract the block data if I can get my hands on a reasonable implementation. The poor-man's alternative is to request block data (which I would then restrict to what's actually visible to the agent) and pretend that I had solved the vision problem. Sure, you can call this cheating, but I guess I am simply more interested in other aspects of the game. Actually, I don't consider the requested chunk as very big :) 10 blocks in each direction is not much if you aim to replace vision. However, it would be enough for reasonable local navigation (without building a map). With a 5x5x5 blocks field of view this is not very reasonable.

[edran]

If you engineer the domain a bit, you can probably reduce the input dimensionality to 21xAx21 where A ~ 3. Or something like that.

This is the case because on the Y axis (i.e. the height in the world coordinate system) you probably want to perceive blocks that:

• might hinder your path;
• your agent might want to pick and/or interact with;

all of which should be roughly on the same height level as your agent. Unless you want to explore mines or multi-layer constructs, you should be able to get a relatively good amount of information.

[TGlas]

Exactly, that's what I want. However, that would restrict me to relatively flat biomes, otherwise I am back to a very local field of view that does not allow for meaningful path planning, which is what you do when navigating the world as a human player. Of course I don't ever need all 21^3 blocks, however, there is no interface to query only the visible surface blocks.

[timhutton]

Hi Tobias,

Yes, that all makes sense. We'll have to think about how to make the grid observation faster.

Until then, you could experiment with the path planning in different ways. Since you have control over the world you could do path planning around obstacles you create yourself. These wouldn't need grid observations since you know where they are.

Another possibility is to use the depth data. See depth_map_runner.py for an example.

Tim

[TGlas]

Dear Tim, Thanks, your proposal should work at least for testing and debugging purposes. The depth data is an interesting option, I'll think about it. Anyway, I am looking forward to a speedup, since in the end I would like to work with a standard Minecraft world. Tobias

[DaveyBiggers]

Hi Tobias, it turns out there was indeed a "source of inefficiency", which we would not have spotted if you hadn't raised the question. [We were calling DataOutputStream.writeBytes(String), which turns out to be several orders of magnitude slower than converting the string to bytes and calling DataOutputStream.write(byte[])].

I've remedied this, and another bug (issue #52) which was also breaking the GridObserver, and I'm now not seeing any noticeable slowdown in Minecraft behaviour when sending a 21x21x21 grid. In fact, I'm only seeing a slight slowdown when sending 61x21x61 (around 78k blocks).

Hopefully this will give you enough scope for your experiments. There is currently an upper limit on the size which occurs when the amount of data in each observation message exceeds 2^20 bytes - this is a limit in Minecraft's own messaging system. (You will easily hit this if you request, say, 61x61x61 blocks.) If there turns out to be a demand for grids of this size, we will look at ways to solve this problem. In the meantime, I'll close this issue.

• Fixed in 7c19d4323d57af5a2039b75960563047342cd55a

[TGlas]

Thanks for taking care of this. I am eager to check this out, so I'll try to compile the latest repository version. Don't worry about extremely large grids for now.

[TGlas]

Tested and works great. Thanks again!