gnembon / fabric-carpet

Fabric Carpet
MIT License
1.67k stars 261 forks source link

Convolutions #593

Open MaxwellWibert opened 5 years ago

MaxwellWibert commented 5 years ago

It would be very cool to be able to perform convolutions on a selected area with user-specified kernels. Properly implemented, this could be really useful to easily and seamlessly smooth out landscapes (using Gaussian kernels), generate Fourier series that generate the describe the terrain (using Dirichlet or Feijer Kernels), or locate block patterns in the map (using a kernel that the user designs with blocks in-game).

MaxwellWibert commented 5 years ago

I'm in a busy stretch in school and I would need some time to learn the Scarpit API before I could submit a pull request. If you want to work on it before then, I would suggest a few ideas on implementation:

As a purely mathematical concept, convolution is basically the process of taking some map called the "kernel" and scanning it across the values of some input map block by block.

For example, suppose we wanted to compute a 1-dimensional kernel on the function f(n) which is defined on every integer, using a kernel g(n) which is defined on every integer from -5 to 5 (again, automatically taking value zero everywhere else). Then we'd compute the convolution as <f,g>(n) = sum_{j=-5}^{j=5} f(n+j)g(j). This has the effect of shifting g over by n blocks and then comparing how similar f is to that shifted copy of g over that area.

In 2 dimensions, we might want map f to be defined on every ordered pair (n,m) where both n and m are integers, and we might want g to be defined on every ordered pair (n,m) where n and m are integers both between -5 and 5. Then we could compute <f,g>(n,m) = sum{j=-5}^{j=5} (sum{k=-5}^{k=5} f(n+j,m+k)g(j,k)). This similarly has the effect of shifting to be centered at (n,m) and seeing how similar it is to f in that region. Generalizing to arbitrary n-dimensional domains for our map and kernel, we simply add more integer coordinates to our map and kernel and sum over a higher dimensional array of indices

So, as for functionality in Minecraft, we have to be able to specify the dimension of the convolution, and somehow define functions and kernels within that dimension. The implementation should be robust enough to return an error if the function/kernel fails to match the dimension specified.

A few example for how the input map might be selected: 1) we could define a 2-d map f on every (x,z) coordinate pair in some specified region by looking at the y coordinate of the highest solid block in that column. 2) define a 2-d map on (x,z) coordinates by counting the number of solid blocks in that column 3) we could define a 3-d map to take a value of 1 everywhere there is a solid block in some region and 0 everywhere else 4) We can generalize any of the three examples above by specifying what kind of blocks we are looking for. Do we want to restrict to solid blocks? spawnable blocks? Diamond ore?

A few examples of how the kernel might be selected: 1) Procedurally generated kernels with user-entered parameters. For example, the user could choose to input a 2-d Gaussian kernel and specify the standard deviation in each direction or "spread" of that Gaussian to control how much they want to smooth out the terrain in each direction. 2) Random noise kernels like Perlin noise, where the user similarly inputs parameters on what scale of detail they want to focus on. 3) Manually input kernels generated based on a user-selected region similar to the examples in the input function section.

I'm sure there are other useful kernels and input functions, but that's all I can think of now. If you want to collaborate on this together drop me a message.

gnembon commented 5 years ago

I am well aware what convolutions are, initially 'diamond' and 'rect' functions were named with conv..., but it might be difficult to represent them nicely for the programmer. The benefit of them would be that currently if scan is applied to an area and blocks are overwritten - this affects blocks evaluated after them in the neighbouring area while with proper convolutions this doesn't happen. Apart from that, the space is not homogeneous - blocks are not just numbers on one scale - as they represent many different things, and convolutions implemented efficiently involve spectral space which would be not that easy to define on a 3d block space. If you think about it, the basic wither cage finder example was kind of 3x3x1 convolution applied on a larger area: scan(c_x, 4, c_z, radius, 0, radius, if(for(rect(_x,_y,z, 1, 0, 1), == 'bedrock')==9, ...

MaxwellWibert commented 5 years ago

@gnembon , I hope I did not offend you. I think as a habit from my math background I tend to err on the side of overexplaining things, but I don't mean to suggest you don't know what they are. This comment was also kind of an open letter. The idea for a generalized convolution function actually hit me when I saw the wither cage finder. And while I agree that the inhomogeneity of Minecraft blocks does mean that the convolution wouldn't be an end-all-be-all tool, I think it could still be worthwhile including in the out-of-the-box utilities. For example, I really think the Gaussian convolution would give a more naturally smooth looking terrain output.

I'm gonna think a bit about how to implement convolutions efficiently in a way that doesn't run into the problem of later loops picking up the changes of earlier loops. If I come up with something, would you be interested in taking a look?

gnembon commented 3 years ago

pobably not gonna happen, or will be added as an extension to scarpet, but hey. Worth keeping.

gnembon commented 3 years ago

I was looking for a light tensor library that is not tensorflow, for linalg operations. Coudn't find any good ones on the java side. maybe there are still some that I don't know of

MaxwellWibert commented 10 months ago

I was looking for a light tensor library that is not tensorflow, for linalg operations. Coudn't find any good ones on the java side. maybe there are still some that I don't know of

https://github.com/Gleethos/neureka may be a good option if you guys still wanted to pursue this for some reason