search

eclipse-corrosion / corrosion

Eclipse Corrosion - Rust edition in Eclipse IDE
Eclipse Public License 2.0
224 stars 31 forks source link

Syntax coloring lost after load or reformat #243

Open norru opened 5 years ago

norru commented 5 years ago

I have observed Rust source syntax coloring loss occasionally, usually after:

Closing and reopening the affected file is an effective, albeit annoying, workaround.

mickaelistria commented 5 years ago

Seems like https://github.com/eclipse/tm4e/issues/203

norru commented 5 years ago

I don't get that exception though.

mickaelistria commented 5 years ago

Try http://download.eclipse.org/tm4e/snapshots/ which should reduce the chances for such problem to happen. It's still not perfect on TM4E side (it's not much tolerant to parallel changes such as editing when highlighting is computed), but it should be bette.r

norru commented 5 years ago

I am already on the latest tm4e snapshot. Doesn't happen in older versions.

mickaelistria commented 5 years ago

Yesterday's build?

norru commented 5 years ago

Uhm, no. Eclipse says nothing new though. Perhaps the build hasn't made it to the update site?

norru commented 5 years ago

Is this the one?

  TextMate Core 0.3.3.201905072152  tm-feature.feature.group    Angelo ZERR
mickaelistria commented 5 years ago

Yes, should be this one. Are you still seeing the issue? As frequently as before?

norru commented 5 years ago

Yep. Pretty much all the time.

mickaelistria commented 5 years ago

Can you please list the version of the various org.eclipse.tm4e plugins you have installed ?

norru commented 5 years ago 
org.eclipse.tm4e.core (0.3.3.201905072119) "TextMate support in Java" [Resolved]
org.eclipse.tm4e.languageconfiguration (0.3.3.201905072147) "TextMate support in Eclipse IDE - Language Configurations" [Resolved]
org.eclipse.tm4e.markdown (0.3.3.201905072152) "TextMate support in Eclipse IDE - Markdown" [Starting]
org.eclipse.tm4e.registry (0.3.3.201905072139) "TextMate support in Eclipse IDE - Registry management" [Resolved]
org.eclipse.tm4e.ui (0.3.3.201905072119) "TextMate support in Eclipse IDE - UI" [Active]
org.eclipse.tm4e.ui.editor (0.2.0.201809031154) "TextMate support in Eclipse IDE - Simple TextMate Editor" [Starting]
norru commented 5 years ago

I have removed this one as it appears to be obsolete, no changes:

org.eclipse.tm4e.ui.editor (0.2.0.201809031154) "TextMate support in Eclipse IDE - Simple TextMate Editor" [Starting]
mickaelistria commented 5 years ago

I tried another crappy hack to TM4E avoid interruption of the syntax highlighting thread in case of error. It's quite ugly, but it may help. Would you like to give it a try?

norru commented 5 years ago

I could give it a shot, but what is exactly the problem? Is it a fundamental problem in how concurrency is handled? Can a "proper" solution for this issue be found for this issue or it is an architectural flaw?

mickaelistria commented 5 years ago

I don't know what is the root issue in your case, but in similar case I see, the issue is that the document changes while TM4E is trying to parse it and attempts to read a part of it that doesn't exist any more. There are for sure more elegant ways to deal with it, but practically, given the amount of workforce available, let's try a dummy one which consists in catching and ignoring such errors.

norru commented 5 years ago

I don't know what is the root issue in your case, the document changes while TM4E is trying to parse it

Uhm. My machine is very fast and powerful. If there is a race condition, it might have one of the branches win more often.

attempts to read a part of it that doesn't exist any more.

Shouldn't TM4E then try again when the document is not changing? How hard it is to add a hotkey to reparse manually? I'm not happy to have to close and reopen each document in the editor every time this happens (which is basically all the time right now).

norru commented 5 years ago

Anyway, if there is something that can mitigate this, I'm ok to try it.

mickaelistria commented 5 years ago

Shouldn't TM4E then try again when the document is not changing?

That's somehow what the last build should implement.

How hard it is to add a hotkey to reparse manually?

It's not hard, but it's a workaround. I don't have enough time to build or review workarounds when there are actual issues to fix ;)

norru commented 5 years ago

@mickaelistria I have got a new TextMate Core snapshot about 15 minutes ago. So far it does look a bit better. Will keep testing throughout the day tomorrow and keep you posted. Thanks for now.

norru commented 5 years ago

Still getting this quite often, about 50% frequency.

I've got two workarounds:

Both are awfully annoying. To be fair, given the bad experience I've had in the last couple of months, I'm planning to give CLion another shot.

norru commented 5 years ago

It's now ~100% again. It probably depends on the source file's length. Formatting in-editor is pretty much useless. I need to close and open editor windows again every single time I hit the format shortcut. If the file is large I lose track.

cargo fmt is the only viable workaround.

norru commented 5 years ago

Problem seems less frequent with Eclipse 2019-06

mickaelistria commented 5 years ago

I'm trying to work on it, but the lack of reproducer is really making it almost impossible to ensure fix works. Can you share an example project and minimal steps to reproduce it?

mickaelistria commented 5 years ago

I've pushed a cleanup patch with some behavior enhancements to tm4e that might help. You can install the new build of TM4E from http://download.eclipse.org/tm4e/snapshots/ and maybe this will help.

norru commented 5 years ago

I'm on Eclipse 2019-06 RC1 if it helps.

Can you share an example project and minimal steps to reproduce it?

Cannot share the project I'm working on but the problem is widespread. I think the problem is more frequent if the affected file is large (>1000 lines?)

norru commented 5 years ago

Still open in Version: 2019-06 (4.12.0) Build id: 20190614-1200

norru commented 5 years ago

If it helps, hitting Ctrl+Shift+F at the beginning of the file makes this issue less frequent than when the shortcut is hit near the bottom of the file.

norru commented 5 years ago

Additional information: once this bug appears, syntax highlight stops working altogether for the affected editor, ie typing new code will always keep the code text black. The only workaround I found is to close and reopen the editor tab.

norru commented 5 years ago

Still open in Corrosion 0.4.2.201907160841

norru commented 5 years ago

@mickaelistria Steps to reproduce:

This is just an example. The problem is widespread and can be easily reproduced by loading any Rust project and attempting to edit any nontrivial file (the file in the example is ~250 lines long). I could not reproduce on very small files (<50 lines).

The problem may well be timing related so CPU speed may play a role (perhaps it's harder to reproduce in slower machines?).

mickaelistria commented 5 years ago

Can't reproduce here. I agree it seems like CPU performance have an impact. Can you please try https://hudson.eclipse.org/tm4e/job/snapshot/ ? An interesting patch was merged earlier today for a probably similar issue with Dart, so maybe the patch improves the state for Corrosion as well.

norru commented 5 years ago

@mickaelistria I have got TextMate Core 0.3.4.201907151606 tm-feature.feature.group Angelo ZERR earlier today, does seem to be matching the latest build.

With this version I am no longer able to reproduce by simply adding/removing empty lines. However, the following would:

  1. clone and import https://github.com/itadinanta/rust-oids.git
  2. open src/app/main.rs in a Rust editor
  3. go to line 30
  4. add two tabs at the beginning of the line.
  5. Ctrl+Shift+F
  6. Repeat from 4 a few times

For some reasons, the bug now appears only if the formatting does not change the length of the file (number of lines).

In fact artificially adding a line before formatting could be a better workaround than close/reopen. EDIT: I have observed the bad behaviour again even if the formatting does not change the line count.

norru commented 5 years ago

Update: the workaround detailed above makes https://github.com/eclipse/corrosion/issues/261 more frequent.

norru commented 5 years ago

Still open on 2019-08-12.

norru commented 5 years ago

Still open on 2019-08-21 (after a flurry of lsp4j snapshot updates).

mickaelistria commented 5 years ago

LSP4J or LSP4E aren't involved here, as mentioned in https://github.com/eclipse/corrosion/issues/243#issuecomment-488701905 only TM4E is to be fixed. I'm still not able to reproduce this issue deterministically, so not able to fix it. I would suggest anyone interested in fixing it to get into the discussion on the TM4E ticket linked above.

norru commented 5 years ago

Still open on 2019-09-26

norru commented 5 years ago

Problem still visible but with reduced frequency on 2019-10-31

norru commented 4 years ago

Still open on 2020-01-28, with high frequency.

norru commented 4 years ago

As suggested by @mickaelistria I have managed to reproduce this with basically no effort (except for the SDK setup itself) in the Eclipse SDK.

I am puzzled by the difficulty other people are encountering when attempting to reproduce this.

I understand this bug can not be reproduced deterministically but this is its nature - it's probably a race condition which by nature cannot be triggered reliably. However, this bug is very frequent and very easy to reproduce.

image

Anyway, I have no idea how to proceed from here to track it :)

Besides, I have attempted to load in the tm4e and lsp4e projects but I am not able to get a clean build in the SDK. I can only work with Corrosion on its own.

image @mickaelistria @Boereck Any suggestions?

mickaelistria commented 4 years ago

Any suggestions?

To what precisely?

norru commented 4 years ago

I need help on general directions on what the next steps could be in order to pinpoint the problem. I have no idea on where to start looking/tracing/placing breakpoints.

First of all, do I require to have the source code projects for Corrosion, lsp4e and tm4e all together? If yes, first of all I need to figure out how to get rid of all the errors. I've tried the target-platform files for all 3 projects but none produces a clear result.

I've got latest master of all 3.

mickaelistria commented 4 years ago

I think the simplest is to

  1. Have only TM4E sources in your workspace
  2. Build your own target-platform which contains Corrosion snapshots

Then when doing "Run As > Eclipse application", you'll be able to put breakpoint into TM4E classes. TM4E has some troubleshooting hints that may help you in this noble quest: https://github.com/eclipse/tm4e/blob/master/documentation/TROUBLESHOOTING.md

norru commented 4 years ago

Enabling the settings above produced this dump while attempting to reproduce the problem. EDIT: the dump is generated on editor close, no dump produced when the error actually occurs.


package org.eclipse.tm4e.ui.text;

import org.eclipse.jface.text.Document;
import org.eclipse.jface.text.IDocument;
import org.eclipse.jface.text.TextViewer;
import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;
import org.eclipse.tm4e.core.grammar.IGrammar;
import org.eclipse.tm4e.core.registry.Registry;
import org.eclipse.tm4e.ui.text.TMPresentationReconciler;
import org.eclipse.tm4e.ui.themes.ITokenProvider;
import org.eclipse.tm4e.ui.themes.css.CSSTokenProvider;
import org.junit.Test;

public class TMPresentationReconcilerTest {

    @Test
    public void colorize() throws Exception {

        Display display = new Display();
        Shell shell = new Shell(display);
        TextViewer viewer = new TextViewer(shell, SWT.NONE);
        IDocument document = new Document();
        viewer.setDocument(document);
        document.set("use cgmath;\nuse core::resource;\nuse frontend::render::formats;\nuse frontend::render::RenderFactoryExt;\nuse frontend::render::Result;\nuse frontend::render::Style;\nuse gfx;\nuse gfx::state::ColorMask;\nuse gfx::state::{Blend, BlendChannel, BlendValue, Equation, Factor};\nuse gfx::traits::FactoryExt;\nuse std::result;\n\npub type PrimitiveIndex = i16;\npub type VertexIndex = u16;\n\ngfx_vertex_struct!(VertexPosNormal {\n   pos: [f32; 3] = \"a_Pos\",\n    normal: [f32; 3] = \"a_Normal\",\n  tangent: [f32; 3] = \"a_Tangent\",\n    tex_coord: [f32; 2] = \"a_TexCoord\",\n primitive_index: PrimitiveIndex = \"a_PrimIndex\",\n});\n\npub type Vertex = VertexPosNormal;\n\nmacro_rules! new_vertex {\n    ($pos:expr, $tex_coord:expr) => {\n     Vertex { pos: $pos, normal: [0., 0., 1.], tangent: [1., 0., 0.], tex_coord: $tex_coord, primitive_index: 0 }\n  };\n}\n\nimpl Default for VertexPosNormal {\n   fn default() -> Self { new_vertex!([0.; 3], [0.5, 0.5]) }\n}\n\nimpl VertexPosNormal {\n    pub fn new(pos: [f32; 3], tex_coord: [f32; 2]) -> VertexPosNormal { new_vertex!(pos, tex_coord) }\n}\n\npub type M44 = cgmath::Matrix4<f32>;\n\nconst MAX_NUM_TOTAL_LIGHTS: usize = 16;\nconst MAX_NUM_TOTAL_TRANSFORMS: usize = 256;\n\npub const PREMULT: Blend = Blend {\n   color: BlendChannel {\n     equation: Equation::Add,\n      source: Factor::One,\n      destination: Factor::OneMinus(BlendValue::SourceAlpha),\n   },\n    alpha: BlendChannel { equation: Equation::Add, source: Factor::One, destination: Factor::One },\n};\n\ngfx_defines!(\n  constant PointLight {\n     propagation: [f32; 4] = \"propagation\",\n      center: [f32; 4] = \"center\",\n        color: [f32; 4] = \"color\",\n  }\n\n   constant CameraArgs {\n     proj: [[f32; 4]; 4] = \"u_Proj\",\n     view: [[f32; 4]; 4] = \"u_View\",\n }\n\n   constant ModelArgs {\n      transform: [[f32; 4]; 4] = \"transform\",\n }\n\n   constant FragmentArgs {\n       light_count: i32 = \"u_LightCount\",\n  }\n\n   constant MaterialArgs {\n       emissive: [f32; 4] = \"u_Emissive\",\n      effect: [f32; 4] = \"u_Effect\",\n  }\n\n   pipeline shaded {\n     vbuf: gfx::VertexBuffer < VertexPosNormal > = (),\n     camera_args: gfx::ConstantBuffer < CameraArgs > = \"cb_CameraArgs\",\n      model_args: gfx::ConstantBuffer < ModelArgs> = \"u_ModelArgs\",\n       fragment_args: gfx::ConstantBuffer <FragmentArgs > = \"cb_FragmentArgs\",\n     material_args: gfx::ConstantBuffer< MaterialArgs > = \"cb_MaterialArgs\",\n     lights: gfx::ConstantBuffer < PointLight > = \"u_Lights\",\n        color_target: gfx::BlendTarget <formats::RenderColorFormat> = (\"o_Color\", ColorMask::all(), PREMULT),\n   depth_target: gfx::DepthTarget <formats::RenderDepthFormat> = gfx::preset::depth::LESS_EQUAL_WRITE,\n   }\n /*\n        pipeline blend {\n          vbuf: gfx::VertexBuffer<VertexPosNormal> = (),\n            camera_args: gfx::ConstantBuffer<CameraArgs> = \"cb_CameraArgs\",\n         model_args: gfx::ConstantBuffer<ModelArgs> = \"cb_ModelArgs\",\n            fragment_args: gfx::ConstantBuffer<FragmentArgs> = \"cb_FragmentArgs\",\n           material_args: gfx::ConstantBuffer<MaterialArgs> = \"cb_MaterialArgs\",\n           lights: gfx::ConstantBuffer<PointLight> = \"u_Lights\",\n           color_target: gfx::BlendTarget<formats::RenderColorFormat> = (\"o_Color\", gfx::state::MASK_ALL, gfx::preset::blend::ALPHA),\n          depth_target: gfx::DepthTarget<formats::RenderDepthFormat> = gfx::preset::depth::LESS_EQUAL_WRITE,\n        }\n */\n\n);\n\npub type ShadedInit<'f> = shaded::Init<'f>;\n\nuse std::marker::PhantomData;\n\npub struct ForwardLighting<R: gfx::Resources, C: gfx::CommandBuffer<R>, D>\nwhere D: gfx::pso::PipelineInit {\n camera: gfx::handle::Buffer<R, CameraArgs>,\n   model: gfx::handle::Buffer<R, ModelArgs>,\n fragment: gfx::handle::Buffer<R, FragmentArgs>,\n   material: gfx::handle::Buffer<R, MaterialArgs>,\n   lights: gfx::handle::Buffer<R, PointLight>,\n   pso: [gfx::pso::PipelineState<R, D::Meta>; Style::Count as usize],\n    _buffer: PhantomData<C>,\n}\n\nimpl<R: gfx::Resources, C: gfx::CommandBuffer<R>, D> ForwardLighting<R, C, D>\nwhere D: gfx::pso::PipelineInit + Clone\n{\n  pub fn new<F>(\n        factory: &mut F,\n      res: &dyn resource::ResourceLoader<u8>,\n       init: D,\n  ) -> Result<ForwardLighting<R, C, D>>\n where\n     F: gfx::Factory<R>,\n   {\n     let lights = factory.create_constant_buffer(MAX_NUM_TOTAL_LIGHTS);\n        let camera = factory.create_constant_buffer(1);\n       let fragment = factory.create_constant_buffer(1);\n     let model = factory.create_constant_buffer(MAX_NUM_TOTAL_TRANSFORMS);\n     let material = factory.create_constant_buffer(MAX_NUM_TOTAL_TRANSFORMS);\n\n        macro_rules! load_shaders {\n           ($v:expr, $f:expr) => {\n               factory.create_shader_set(\n                    &res.load(concat!(\"shaders/forward/\", $v, \".vert\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $f, \".frag\"))?,\n                 )\n         };\n\n          ($g:expr, $v:expr, $f:expr) => {\n              factory.create_shader_set_with_geometry(\n                  &res.load(concat!(\"shaders/forward/\", $g, \".geom\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $v, \".vert\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $f, \".frag\"))?,\n                 )\n         };\n        };\n\n      let flat_shaders = load_shaders!(\"lighting\", \"lighting_flat\")?;\n       let wire_shaders = load_shaders!(\"lighting\", \"lighting_poly\")?;\n       let solid_shaders = load_shaders!(\"triangle_edge\", \"lighting\", \"lighting_poly\")?;\n       let stage_shaders = load_shaders!(\"lighting\", \"lighting_stage\")?;\n     let particle_shaders = load_shaders!(\"unlit\", \"ripple_particle\")?;\n        let ball_shaders = load_shaders!(\"point_ball\", \"lighting\", \"lighting_poly\")?;\n\n     let solid_rasterizer =\n            gfx::state::Rasterizer { samples: Some(gfx::state::MultiSample), ..gfx::state::Rasterizer::new_fill() };\n\n        let line_rasterizer = gfx::state::Rasterizer { method: gfx::state::RasterMethod::Line(2), ..solid_rasterizer };\n       let debug_line_rasterizer =\n           gfx::state::Rasterizer { method: gfx::state::RasterMethod::Line(1), ..solid_rasterizer };\n\n       let ball_pso =\n            Self::new_pso(factory, &ball_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n     let poly_pso =\n            Self::new_pso(factory, &solid_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let stage_pso =\n           Self::new_pso(factory, &stage_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let particle_pso =\n            Self::new_pso(factory, &particle_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n     let wireframe_pso =\n           Self::new_pso(factory, &wire_shaders, gfx::Primitive::TriangleList, line_rasterizer, init.clone())?;\n      let lit_pso =\n         Self::new_pso(factory, &solid_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let lines_pso = Self::new_pso(factory, &flat_shaders, gfx::Primitive::LineList, line_rasterizer, init.clone())?;\n      let debug_lines_pso =\n         Self::new_pso(factory, &flat_shaders, gfx::Primitive::LineList, debug_line_rasterizer, init)?;\n        Ok(ForwardLighting {\n          camera,\n           model,\n            fragment,\n         material,\n         lights,\n           pso: [ball_pso, poly_pso, stage_pso, particle_pso, wireframe_pso, lit_pso, lines_pso, debug_lines_pso],\n           _buffer: PhantomData,\n     })\n    }\n\n   fn new_pso<F>(\n        factory: &mut F,\n      shaders: &gfx::ShaderSet<R>,\n      primitive: gfx::Primitive,\n        rasterizer: gfx::state::Rasterizer,\n       init: D,\n  ) -> result::Result<gfx::pso::PipelineState<R, D::Meta>, gfx::PipelineStateError<String>>\n where\n     F: gfx::Factory<R>,\n   {\n     factory.create_pipeline_state(&shaders, primitive, rasterizer, init)\n  }\n\n   pub fn setup(\n     &self,\n        encoder: &mut gfx::Encoder<R, C>,\n     camera_projection: M44,\n   camera_view: M44,\n     lights: &[PointLight],\n    ) -> Result<()>\n   {\n     let mut lights_buf = lights.to_owned();\n\n     let count = lights_buf.len();\n     while lights_buf.len() < MAX_NUM_TOTAL_LIGHTS {\n           lights_buf.push(PointLight {\n              propagation: [0., 0., 0., 0.],\n                color: [0., 0., 0., 0.],\n              center: [0., 0., 0., 0.],\n         })\n        }\n\n       encoder.update_buffer(&self.lights, &lights_buf[..], 0)?;\n     encoder.update_constant_buffer(&self.camera, &CameraArgs {\n            proj: camera_projection.into(),\n           view: camera_view.into(),\n     });\n       encoder.update_constant_buffer(&self.fragment, &FragmentArgs { light_count: count as i32 });\n      Ok(())\n    }\n}\n\nimpl<R: gfx::Resources, C: gfx::CommandBuffer<R>> ForwardLighting<R, C, shaded::Init<'static>> {\n#[allow(clippy::too_many_arguments)]\n    pub fn draw_primitives(\n       &self,\n        shader: Style,\n        encoder: &mut gfx::Encoder<R, C>,\n     vertices: gfx::handle::Buffer<R, VertexPosNormal>,\n        indices: &gfx::Slice<R>,\n      models: &[ModelArgs],\n     materials: &[MaterialArgs],\n       color_buffer: &gfx::handle::RenderTargetView<R, formats::RenderColorFormat>,\n      depth_buffer: &gfx::handle::DepthStencilView<R, formats::RenderDepthFormat>,\n  ) -> Result<()>\n   {\n     encoder.update_buffer(&self.model, &models, 0)?;\n      encoder.update_buffer(&self.material, &materials, 0)?;\n        encoder.draw(indices, &self.pso[shader as usize], &shaded::Data {\n         vbuf: vertices,\n   fragment_args: self.fragment.clone(),\n         material_args: self.material.clone(),\n         camera_args: self.camera.clone(),\n         model_args: self.model.clone(),\n   lights: self.lights.clone(),\n          color_target: color_buffer.clone(),\n           depth_target: depth_buffer.clone(),\n       });\n       Ok(())\n    }\n}\n");

        TMPresentationReconciler reconciler = new TMPresentationReconciler();
        reconciler.setTokenProvider(getTokenProvider());
        reconciler.setGrammar(getGrammar());
        reconciler.install(viewer);

        document.replace(7126, 0, "\n");
        document.replace(7127, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7127, 2, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7127, 0, "\n");
        document.replace(7128, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7127, 2, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7840, 0, "\n");
        document.replace(7841, 0, "\n");
        document.replace(7842, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7841, 3, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7539, 0, "\n");
        document.replace(7540, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7540, 2, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7278, 0, " ");
        document.replace(7279, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(7278, 44, "        let mut lights_buf = lights.to_owned();\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(8142, 0, "\n       ");
        document.replace(8145, 0, "\n       ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(8143, 6, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6465, 0, "\n");
        document.replace(6466, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6465, 2, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6563, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6563, 24, "        Ok(ForwardLighting {\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6564, 0, "\n   ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6563, 2, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6564, 0, " ");
        document.replace(6565, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6563, 25, "        Ok(ForwardLighting {\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6328, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(6326, 116, "       let lines_pso = Self::new_pso(factory, &flat_shaders, gfx::Primitive::LineList, line_rasterizer, init.clone())?;\n");
        viewer.invalidateTextPresentation(0, 0);
        viewer.invalidateTextPresentation(166, 8209);
        document.replace(3686, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(3678, 79, "impl<R: gfx::Resources, C: gfx::CommandBuffer<R>, D> ForwardLighting<R, C, D>\n");
        viewer.invalidateTextPresentation(0, 0);
        viewer.invalidateTextPresentation(3687, 22);
        viewer.invalidateTextPresentation(166, 8209);
        document.replace(3960, 0, " ");
        viewer.invalidateTextPresentation(166, 8210);
        document.replace(3960, 1, "");
        document.replace(3959, 0, " ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(3957, 70, "        let lights = factory.create_constant_buffer(MAX_NUM_TOTAL_LIGHTS);\n");
        viewer.invalidateTextPresentation(0, 0);

        while (!shell.isDisposed()) {
        }
    }
    private static ITokenProvider getTokenProvider() {
        return new CSSTokenProvider(TMPresentationReconcilerTest.class.getResourceAsStream("Solarized-light.css"));
    }

    private static IGrammar getGrammar() {
        Registry registry = new Registry();
        try {
        String grammar="YouGrammar.tmLanguage";
            return registry.loadGrammarFromPathSync(grammar,TMPresentationReconcilerTest.class.getResourceAsStream(grammar));
        } catch (Exception e) {
            e.printStackTrace();
        return null;
        }
    }
}
norru commented 4 years ago

Another example:

package org.eclipse.tm4e.ui.text;

import org.eclipse.jface.text.Document;
import org.eclipse.jface.text.IDocument;
import org.eclipse.jface.text.TextViewer;
import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;
import org.eclipse.tm4e.core.grammar.IGrammar;
import org.eclipse.tm4e.core.registry.Registry;
import org.eclipse.tm4e.ui.text.TMPresentationReconciler;
import org.eclipse.tm4e.ui.themes.ITokenProvider;
import org.eclipse.tm4e.ui.themes.css.CSSTokenProvider;
import org.junit.Test;

public class TMPresentationReconcilerTest {

    @Test
    public void colorize() throws Exception {

        Display display = new Display();
        Shell shell = new Shell(display);
        TextViewer viewer = new TextViewer(shell, SWT.NONE);
        IDocument document = new Document();
        viewer.setDocument(document);
        document.set("use cgmath;\nuse core::resource;\nuse frontend::render::formats;\nuse frontend::render::RenderFactoryExt;\nuse frontend::render::Result;\nuse frontend::render::Style;\nuse gfx;\nuse gfx::state::ColorMask;\nuse gfx::state::{Blend, BlendChannel, BlendValue, Equation, Factor};\nuse gfx::traits::FactoryExt;\nuse std::result;\n\npub type PrimitiveIndex = i16;\npub type VertexIndex = u16;\n\ngfx_vertex_struct!(VertexPosNormal {\n   pos: [f32; 3] = \"a_Pos\",\n    normal: [f32; 3] = \"a_Normal\",\n  tangent: [f32; 3] = \"a_Tangent\",\n    tex_coord: [f32; 2] = \"a_TexCoord\",\n primitive_index: PrimitiveIndex = \"a_PrimIndex\",\n});\n\npub type Vertex = VertexPosNormal;\n\nmacro_rules! new_vertex {\n    ($pos:expr, $tex_coord:expr) => {\n     Vertex { pos: $pos, normal: [0., 0., 1.], tangent: [1., 0., 0.], tex_coord: $tex_coord, primitive_index: 0 }\n  };\n}\n\nimpl Default for VertexPosNormal {\n   fn default() -> Self { new_vertex!([0.; 3], [0.5, 0.5]) }\n}\n\nimpl VertexPosNormal {\n    pub fn new(pos: [f32; 3], tex_coord: [f32; 2]) -> VertexPosNormal { new_vertex!(pos, tex_coord) }\n}\n\npub type M44 = cgmath::Matrix4<f32>;\n\nconst MAX_NUM_TOTAL_LIGHTS: usize = 16;\nconst MAX_NUM_TOTAL_TRANSFORMS: usize = 256;\n\npub const PREMULT: Blend = Blend {\n   color: BlendChannel {\n     equation: Equation::Add,\n      source: Factor::One,\n      destination: Factor::OneMinus(BlendValue::SourceAlpha),\n   },\n    alpha: BlendChannel { equation: Equation::Add, source: Factor::One, destination: Factor::One },\n};\n\ngfx_defines!(\n  constant PointLight {\n     propagation: [f32; 4] = \"propagation\",\n      center: [f32; 4] = \"center\",\n        color: [f32; 4] = \"color\",\n  }\n\n   constant CameraArgs {\n     proj: [[f32; 4]; 4] = \"u_Proj\",\n     view: [[f32; 4]; 4] = \"u_View\",\n }\n\n   constant ModelArgs {\n      transform: [[f32; 4]; 4] = \"transform\",\n }\n\n   constant FragmentArgs {\n       light_count: i32 = \"u_LightCount\",\n  }\n\n   constant MaterialArgs {\n       emissive: [f32; 4] = \"u_Emissive\",\n      effect: [f32; 4] = \"u_Effect\",\n  }\n\n   pipeline shaded {\n     vbuf: gfx::VertexBuffer < VertexPosNormal > = (),\n     camera_args: gfx::ConstantBuffer < CameraArgs > = \"cb_CameraArgs\",\n      model_args: gfx::ConstantBuffer < ModelArgs> = \"u_ModelArgs\",\n       fragment_args: gfx::ConstantBuffer <FragmentArgs > = \"cb_FragmentArgs\",\n     material_args: gfx::ConstantBuffer< MaterialArgs > = \"cb_MaterialArgs\",\n     lights: gfx::ConstantBuffer < PointLight > = \"u_Lights\",\n        color_target: gfx::BlendTarget <formats::RenderColorFormat> = (\"o_Color\", ColorMask::all(), PREMULT),\n   depth_target: gfx::DepthTarget <formats::RenderDepthFormat> = gfx::preset::depth::LESS_EQUAL_WRITE,\n   }\n /*\n        pipeline blend {\n          vbuf: gfx::VertexBuffer<VertexPosNormal> = (),\n            camera_args: gfx::ConstantBuffer<CameraArgs> = \"cb_CameraArgs\",\n         model_args: gfx::ConstantBuffer<ModelArgs> = \"cb_ModelArgs\",\n            fragment_args: gfx::ConstantBuffer<FragmentArgs> = \"cb_FragmentArgs\",\n           material_args: gfx::ConstantBuffer<MaterialArgs> = \"cb_MaterialArgs\",\n           lights: gfx::ConstantBuffer<PointLight> = \"u_Lights\",\n           color_target: gfx::BlendTarget<formats::RenderColorFormat> = (\"o_Color\", gfx::state::MASK_ALL, gfx::preset::blend::ALPHA),\n          depth_target: gfx::DepthTarget<formats::RenderDepthFormat> = gfx::preset::depth::LESS_EQUAL_WRITE,\n        }\n */\n\n);\n\npub type ShadedInit<'f> = shaded::Init<'f>;\n\nuse std::marker::PhantomData;\n\npub struct ForwardLighting<R: gfx::Resources, C: gfx::CommandBuffer<R>, D>\nwhere D: gfx::pso::PipelineInit {\n camera: gfx::handle::Buffer<R, CameraArgs>,\n   model: gfx::handle::Buffer<R, ModelArgs>,\n fragment: gfx::handle::Buffer<R, FragmentArgs>,\n   material: gfx::handle::Buffer<R, MaterialArgs>,\n   lights: gfx::handle::Buffer<R, PointLight>,\n   pso: [gfx::pso::PipelineState<R, D::Meta>; Style::Count as usize],\n    _buffer: PhantomData<C>,\n}\n\nimpl<R: gfx::Resources, C: gfx::CommandBuffer<R>, D> ForwardLighting<R, C, D>\nwhere D: gfx::pso::PipelineInit + Clone\n{\n  pub fn new<F>(\n        factory: &mut F,\n      res: &dyn resource::ResourceLoader<u8>,\n       init: D,\n  ) -> Result<ForwardLighting<R, C, D>>\n where\n     F: gfx::Factory<R>,\n   {\n     let lights = factory.create_constant_buffer(MAX_NUM_TOTAL_LIGHTS);\n        let camera = factory.create_constant_buffer(1);\n       let fragment = factory.create_constant_buffer(1);\n     let model = factory.create_constant_buffer(MAX_NUM_TOTAL_TRANSFORMS);\n     let material = factory.create_constant_buffer(MAX_NUM_TOTAL_TRANSFORMS);\n\n        macro_rules! load_shaders {\n           ($v:expr, $f:expr) => {\n               factory.create_shader_set(\n                    &res.load(concat!(\"shaders/forward/\", $v, \".vert\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $f, \".frag\"))?,\n                 )\n         };\n\n          ($g:expr, $v:expr, $f:expr) => {\n              factory.create_shader_set_with_geometry(\n                  &res.load(concat!(\"shaders/forward/\", $g, \".geom\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $v, \".vert\"))?,\n                 &res.load(concat!(\"shaders/forward/\", $f, \".frag\"))?,\n                 )\n         };\n        };\n\n      let flat_shaders = load_shaders!(\"lighting\", \"lighting_flat\")?;\n       let wire_shaders = load_shaders!(\"lighting\", \"lighting_poly\")?;\n       let solid_shaders = load_shaders!(\"triangle_edge\", \"lighting\", \"lighting_poly\")?;\n       let stage_shaders = load_shaders!(\"lighting\", \"lighting_stage\")?;\n     let particle_shaders = load_shaders!(\"unlit\", \"ripple_particle\")?;\n        let ball_shaders = load_shaders!(\"point_ball\", \"lighting\", \"lighting_poly\")?;\n\n     let solid_rasterizer =\n            gfx::state::Rasterizer { samples: Some(gfx::state::MultiSample), ..gfx::state::Rasterizer::new_fill() };\n\n        let line_rasterizer = gfx::state::Rasterizer { method: gfx::state::RasterMethod::Line(2), ..solid_rasterizer };\n       let debug_line_rasterizer =\n           gfx::state::Rasterizer { method: gfx::state::RasterMethod::Line(1), ..solid_rasterizer };\n\n       let ball_pso =\n            Self::new_pso(factory, &ball_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n     let poly_pso =\n            Self::new_pso(factory, &solid_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let stage_pso =\n           Self::new_pso(factory, &stage_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let particle_pso =\n            Self::new_pso(factory, &particle_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n     let wireframe_pso =\n           Self::new_pso(factory, &wire_shaders, gfx::Primitive::TriangleList, line_rasterizer, init.clone())?;\n      let lit_pso =\n         Self::new_pso(factory, &solid_shaders, gfx::Primitive::TriangleList, solid_rasterizer, init.clone())?;\n        let lines_pso = Self::new_pso(factory, &flat_shaders, gfx::Primitive::LineList, line_rasterizer, init.clone())?;\n      let debug_lines_pso =\n         Self::new_pso(factory, &flat_shaders, gfx::Primitive::LineList, debug_line_rasterizer, init)?;\n        Ok(ForwardLighting {\n          camera,\n           model,\n            fragment,\n         material,\n         lights,\n           pso: [ball_pso, poly_pso, stage_pso, particle_pso, wireframe_pso, lit_pso, lines_pso, debug_lines_pso],\n           _buffer: PhantomData,\n     })\n    }\n\n   fn new_pso<F>(\n        factory: &mut F,\n      shaders: &gfx::ShaderSet<R>,\n      primitive: gfx::Primitive,\n        rasterizer: gfx::state::Rasterizer,\n       init: D,\n  ) -> result::Result<gfx::pso::PipelineState<R, D::Meta>, gfx::PipelineStateError<String>>\n where\n     F: gfx::Factory<R>,\n   {\n     factory.create_pipeline_state(&shaders, primitive, rasterizer, init)\n  }\n\n   pub fn setup(\n     &self,\n        encoder: &mut gfx::Encoder<R, C>,\n     camera_projection: M44,\n   camera_view: M44,\n     lights: &[PointLight],\n    ) -> Result<()>\n   {\n     let mut lights_buf = lights.to_owned();\n\n     let count = lights_buf.len();\n     while lights_buf.len() < MAX_NUM_TOTAL_LIGHTS {\n           lights_buf.push(PointLight {\n              propagation: [0., 0., 0., 0.],\n                color: [0., 0., 0., 0.],\n              center: [0., 0., 0., 0.],\n         })\n        }\n\n       encoder.update_buffer(&self.lights, &lights_buf[..], 0)?;\n     encoder.update_constant_buffer(&self.camera, &CameraArgs {\n            proj: camera_projection.into(),\n           view: camera_view.into(),\n     });\n       encoder.update_constant_buffer(&self.fragment, &FragmentArgs { light_count: count as i32 });\n      Ok(())\n    }\n}\n\nimpl<R: gfx::Resources, C: gfx::CommandBuffer<R>> ForwardLighting<R, C, shaded::Init<'static>> {\n#[allow(clippy::too_many_arguments)]\n    pub fn draw_primitives(\n       &self,\n        shader: Style,\n        encoder: &mut gfx::Encoder<R, C>,\n     vertices: gfx::handle::Buffer<R, VertexPosNormal>,\n        indices: &gfx::Slice<R>,\n      models: &[ModelArgs],\n     materials: &[MaterialArgs],\n       color_buffer: &gfx::handle::RenderTargetView<R, formats::RenderColorFormat>,\n      depth_buffer: &gfx::handle::DepthStencilView<R, formats::RenderDepthFormat>,\n  ) -> Result<()>\n   {\n     encoder.update_buffer(&self.model, &models, 0)?;\n      encoder.update_buffer(&self.material, &materials, 0)?;\n        encoder.draw(indices, &self.pso[shader as usize], &shaded::Data {\n         vbuf: vertices,\n   fragment_args: self.fragment.clone(),\n         material_args: self.material.clone(),\n         camera_args: self.camera.clone(),\n         model_args: self.model.clone(),\n   lights: self.lights.clone(),\n          color_target: color_buffer.clone(),\n           depth_target: depth_buffer.clone(),\n       });\n       Ok(())\n    }\n}\n");

        TMPresentationReconciler reconciler = new TMPresentationReconciler();
        reconciler.setTokenProvider(getTokenProvider());
        reconciler.setGrammar(getGrammar());
        reconciler.install(viewer);

        document.replace(809, 0, "\n");
        document.replace(810, 0, "\n");
        document.replace(811, 0, "\n");
        document.replace(812, 0, "\n");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(810, 4, "");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(907, 0, "  ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(907, 24, "impl VertexPosNormal {\n");
        viewer.invalidateTextPresentation(0, 0);

        while (!shell.isDisposed()) {
        }
    }
    private static ITokenProvider getTokenProvider() {
        return new CSSTokenProvider(TMPresentationReconcilerTest.class.getResourceAsStream("Solarized-light.css"));
    }

    private static IGrammar getGrammar() {
        Registry registry = new Registry();
        try {
        String grammar="YouGrammar.tmLanguage";
            return registry.loadGrammarFromPathSync(grammar,TMPresentationReconcilerTest.class.getResourceAsStream(grammar));
        } catch (Exception e) {
            e.printStackTrace();
        return null;
        }
    }
}
norru commented 4 years ago

Here's a matching error/dump. Error reproduced by:

image


package org.eclipse.tm4e.ui.text;

import org.eclipse.jface.text.Document;
import org.eclipse.jface.text.IDocument;
import org.eclipse.jface.text.TextViewer;
import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;
import org.eclipse.tm4e.core.grammar.IGrammar;
import org.eclipse.tm4e.core.registry.Registry;
import org.eclipse.tm4e.ui.text.TMPresentationReconciler;
import org.eclipse.tm4e.ui.themes.ITokenProvider;
import org.eclipse.tm4e.ui.themes.css.CSSTokenProvider;
import org.junit.Test;

public class TMPresentationReconcilerTest {

    @Test
    public void colorize() throws Exception {

        Display display = new Display();
        Shell shell = new Shell(display);
        TextViewer viewer = new TextViewer(shell, SWT.NONE);
        IDocument document = new Document();
        viewer.setDocument(document);
        document.set("use core::color;\nuse gfx;\n\npub type Rgba = color::Rgba<f32>;\npub type Float4 = [f32; 4];\npub type Float = f32;\npub type RenderColorChannels = gfx::format::R16_G16_B16_A16;\npub type RenderColorFormat = (RenderColorChannels, gfx::format::Float);\npub type RenderDepthFormat = gfx::format::Depth;\n\npub type ScreenColorChannels = gfx::format::R8_G8_B8_A8;\n// Srgba8 broken on Linux\npub type ScreenColorFormat = (ScreenColorChannels, gfx::format::Unorm);\n// Srgba8 broken on Linux\npub type ScreenDepthFormat = gfx::format::Depth;\n\npub type RenderSurface<R> = (\n  gfx::handle::Texture<R, RenderColorChannels>,\n gfx::handle::ShaderResourceView<R, Float4>,\n   gfx::handle::RenderTargetView<R, RenderColorFormat>,\n);\n\npub type DepthSurface<R> = (\n  gfx::handle::Texture<R, gfx::format::D24>,\n    gfx::handle::ShaderResourceView<R, Float>,\n    gfx::handle::DepthStencilView<R, RenderDepthFormat>,\n);\n\npub type RenderSurfaceWithDepth<R> = (\n    gfx::handle::ShaderResourceView<R, Float4>,\n   gfx::handle::RenderTargetView<R, RenderColorFormat>,\n  gfx::handle::DepthStencilView<R, RenderDepthFormat>,\n);\n\npub const MSAA_MODE: gfx::texture::AaMode = gfx::texture::AaMode::Multi(4);\n");

        TMPresentationReconciler reconciler = new TMPresentationReconciler();
        reconciler.setTokenProvider(getTokenProvider());
        reconciler.setGrammar(getGrammar());
        reconciler.install(viewer);

        document.replace(705, 0, "  ");
        viewer.invalidateTextPresentation(0, 0);
        document.replace(705, 30, "pub type DepthSurface<R> = (\n");
        viewer.invalidateTextPresentation(0, 0);

        while (!shell.isDisposed()) {
        }
    }
    private static ITokenProvider getTokenProvider() {
        return new CSSTokenProvider(TMPresentationReconcilerTest.class.getResourceAsStream("Solarized-light.css"));
    }

    private static IGrammar getGrammar() {
        Registry registry = new Registry();
        try {
        String grammar="YouGrammar.tmLanguage";
            return registry.loadGrammarFromPathSync(grammar,TMPresentationReconcilerTest.class.getResourceAsStream(grammar));
        } catch (Exception e) {
            e.printStackTrace();
        return null;
        }
    }
}
norru commented 3 years ago

Still open with Eclipse 2020-12 and matching Corrosion :(

cplir-c commented 3 years ago

On the new june build of corrosion (1.2.1.202106081156) and eclipse 2020-12 doing Window -> Editor -> Split Editor (Vertical) for the first time since eclipse launch on a 200 line rust file results in the second pane consistently being uncolored. It also happens when I leave the vertical split open and restart eclipse.

mickaelistria commented 3 years ago

On the new june build of corrosion (1.2.1.202106081156) and eclipse 2020-12 doing Window -> Editor -> Split Editor (Vertical) for the first time since eclipse launch on a 200 line rust file results in the second pane consistently being uncolored. It also happens when I leave the vertical split open and restart eclipse.

Please open a separate issue for that as it doesn't seem related to the original bug.

akurtakov commented 2 years ago

There have been many fixes and improvements to tm4e so do you still face the issue with latest versions?