Assets<Image>'s get_mut prevents the texture of a mesh using the Image as a texture to be updated

Selene-Amanita commented 1 year ago

Bevy version

v0.10.1

What you did

Render a camera to an image, use that image as a texture of a mesh, use Assets<Image>'s get_mut on the handle of that image

What went wrong

The texture stops updating

Minimal code to reproduce

Comment/uncomment lines

use bevy::{
    prelude::*,
    render::{
        render_resource::*,
        camera::RenderTarget
    },
};

#[derive(Component)]
struct MainCamera; // Camera that is used to render to the window, not the image

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_startup_system(setup)
        .add_system(rotate_other_camera)
        // comment and uncomment this line
        //.add_system(get_mut_image)
        .run();
}

fn get_mut_image (
    mut images: ResMut<Assets<Image>>,
    camera_query: Query<&Camera, Without<MainCamera>>
) {
    let camera = camera_query.single();
    if let RenderTarget::Image(handle) = camera.target.clone() {
        images.get_mut(&handle.clone()); // This line is the culprit, you can uncomment it too
    }
}

// Turn the texture rendering camera to make the texture update
fn rotate_other_camera(
    time: Res<Time>,
    mut camera_query: Query<&mut Transform, (With<Camera>, Without<MainCamera>)>,
) {
    let mut camera_transform = camera_query.single_mut();
    camera_transform.rotate_around(Vec3::ZERO, Quat::from_axis_angle(Vec3::Y, time.delta_seconds() * 9.))
}

// This is adapted from the Bevy example to render to a texture: https://github.com/bevyengine/bevy/blob/latest/examples/3d/render_to_texture.rs
fn setup(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut images: ResMut<Assets<Image>>,
    mut materials: ResMut<Assets<StandardMaterial>>
) {
    commands.insert_resource(AmbientLight {
        color: Color::WHITE,
        brightness: 0.3,
    });

    // Main camera (displayed in window)
    commands.spawn((
        Camera3dBundle {
            transform: Transform::from_xyz(0.0, 0., 20.0).looking_at(Vec3::ZERO, Vec3::Y),
            ..default()
        },
        MainCamera
    ));

    // Second camera (renders to an image, gets rotated, and the image doesn't update if get_mut_image)
    let size = Extent3d {width: 450, height: 450, ..default()};
    let mut camera_image = Image {
        texture_descriptor: TextureDescriptor {
            label: None,
            size,
            dimension: TextureDimension::D2,
            format: TextureFormat::Bgra8UnormSrgb,
            mip_level_count: 1,
            sample_count: 1,
            usage: TextureUsages::TEXTURE_BINDING
                | TextureUsages::COPY_DST
                | TextureUsages::RENDER_ATTACHMENT,
            view_formats: &[],
        },
        ..default()
    };
    camera_image.resize(size);
    let camera_image = images.add(camera_image);

    commands.spawn(
        Camera3dBundle {
            camera: Camera {
                order: -1,
                target: RenderTarget::Image(camera_image),
                ..default()
            },
            transform: Transform::from_xyz(0., 0., 10.),
            ..default()
        }
    );

    // Object the image is rendered to
    let material_handle = materials.add(StandardMaterial {
        base_color_texture: Some(camera_image.clone()),
        reflectance: 0.02,
        unlit: false,
        ..default()
    });
    commands.spawn((
        PbrBundle {
            mesh: meshes.add(Mesh::from(shape::Cube::new(5.))),
            material: material_handle,
            transform: Transform::from_xyz(0., 6., 0.),
            ..default()
        },
    ));

    // Normal object to have something to look at
    let sphere_mesh = meshes.add(Mesh::from(shape::Cube::new(5.)));
    let debug_material = materials.add(StandardMaterial {
        base_color_texture: Some(images.add(uv_debug_texture())),
        ..default()
    });
    commands.spawn(PbrBundle {
        mesh: sphere_mesh,
        material: debug_material,
        ..default()
    });
}

// This is taken from the 3d_shapes example: https://bevyengine.org/examples/3d/3d-shapes/
pub fn uv_debug_texture() -> Image {
    const TEXTURE_SIZE: usize = 8;

    let mut palette: [u8; 32] = [
        255, 102, 159, 255, 255, 159, 102, 255, 236, 255, 102, 255, 121, 255, 102, 255, 102, 255,
        198, 255, 102, 198, 255, 255, 121, 102, 255, 255, 236, 102, 255, 255,
    ];

    let mut texture_data = [0; TEXTURE_SIZE * TEXTURE_SIZE * 4];
    for y in 0..TEXTURE_SIZE {
        let offset = TEXTURE_SIZE * y * 4;
        texture_data[offset..(offset + TEXTURE_SIZE * 4)].copy_from_slice(&palette);
        palette.rotate_right(4);
    }

    Image::new_fill(
        Extent3d {
            width: TEXTURE_SIZE as u32,
            height: TEXTURE_SIZE as u32,
            depth_or_array_layers: 1,
        },
        TextureDimension::D2,
        &texture_data,
        TextureFormat::Rgba8UnormSrgb,
    )
}

Additional information

Discord threads (potentially) related:

Potential relevant system information

Rust version:
Operating System: Linux Mint MATE 1.26, x11
AdapterInfo { name: "Intel(R) Xe Graphics (TGL GT2)", vendor: 32902, device: 39497, device_type: IntegratedGpu, driver: "Intel open-source Mesa driver", driver_info: "Mesa 22.2.5", backend: Vulkan }

mockersf commented 1 year ago

There are two worlds: main world, and render world.

You create the image in the main world, it's completely black at this point
That image gets extracted from the main world to the render world
In the render world, that image gets overwritten with the camera view
In the render world, that image is rendered on a PbrBundle

What happens when you call Assets<Image>.get_mut() is that the asset is marked as changed, and will be extracted again to the render world. As the actual image was never retrieved from the render world to the main world, this is image is still just black in the main world. Also, the change on the image asset is not propagated to the standard material asset which is the one actually displayed.

What fo you expect to be able to do by calling get_mut on the asset?

Selene-Amanita commented 1 year ago

@mockersf I'm not 100% sure I understood that explaination (by "extracted", do you mean "cloned" and that there are actually two images?) but more details just in case:

This code was originally an example to reproduce another potential bug (but I couldn't reproduce it with it :/)
If you use get_mut only once, the same thing still happens
What I was trying to do using get_mut (in a different way in my original code but still with this setup of rendering to a material) was trying to resize the image, but no matter if I resize or not the same thing happen as long as I use get_mut
I understand that I should probably not do that, or do it another way (how?), but for me to know that I should know that the render word is separate, the image is "there", etc... I feel like I shouldn't have to know that to use an innocent-looking get_mut, it should either not have this behaviour, or it should warn against it somewhere in the documentation somehow and provide an alternative (maybe at RenderTarget::Image or StandardMaterial.base_color_texture depending on which one triggers this render world extraction of the image? And maybe in the render to texture example? And maybe in a shader example using a texture?)

Selene-Amanita commented 11 months ago

In case somebody finds this: it is possible to properly update the image by "touching" the material (materials.get_mut(handle)) too: https://discord.com/channels/691052431525675048/1150171576474554468

This is likely waiting on https://github.com/bevyengine/bevy/pull/5080 to be fixed

bevyengine / bevy