Bug 247647 - Linked Resources & Nested Projects: Multiple editors for same resources, Multiple local histories, Multiple sets of markers, etc...

[cobexer]

Revival of: https://bugs.eclipse.org/bugs/show_bug.cgi?id=247647 please see the Bugzilla for more context

Steps To Reproduce: If you create a Project which links to a file in another Project, and/or create a nested Project over an existing Project. You can end up > with multiple ways to access the same file through the Project Explorer, Navigator, etc. views.

This is still very relevant (or even more relevant) with at least Eclipse 22.03!

The specific circumstances that brought me here: Using a file search I get multiple matches in the same "physical" file:

Project layout: /some gradle root project/.project /some gradle root project/relative-path/some-java-subproject/.project /some gradle root project/relative-path/some-java-subproject/src/main/whatever.java

for any search I get 2 matches: some gradle root project/relative-path/some-java-subproject/src/main/whatever.java (this one is very bad) and some-java-subproject/src/main/whatever.java (this is the only valid one)

Opening the wrong match may have surprising unintuitive consequences: The editor may not function properly because the enclosing project is wrong or lacks a neccesary nature. -> in my case even worse: Eclipse (or the language plugin) added the nature to the "some gradle root project" project and thus corrupted the project "permanently".

Is there ever a valid reason to show resources in any project other than the one with the shortest relative path (more specific project)?

A few things of note (IMO):

• even if the more specific project is closed, eclipse must hide the resource: I closed the project after all!
• Lots of maven projects seem to use a parent pom/project approach - most often the sub projects/modules are nested within the parent project (on disk)
• every Gradle project has it's sub projects nested within the root project and therefore is affected by this issue
• every action tht may open files presents an opportunity for the user to get the wrong file opened and thus experience reduced IDE functionality or bugs.

[humphreygao]

https://github.com/eclipse-platform/eclipse.platform.text/issues/31

[mickaelistria]

Thinking deeper, the issue is that JDT requires 1 project per classpath and vice-versa. This requires resources to be duplicated in case of nested projects. The ultimate solution would be that JDT gets modified to be able to deal with multiple settings/classpath per project without requiring multiple projects; then everything would become much simpler as we wouldn't have to import projects as nested.

[iloveeclipse]

The ultimate solution would be that JDT gets modified to be able to deal with multiple settings/classpath per project

You probably mean, allow classpath per source folder / source folders set. I think it is not realistic, that would require rewrite lot of core/UI code in JDT and all its clients and would be a breaking change. Surely you could do that in clean rewrite of JDT (assuming unlimited time and man power you will need), but surely not given existing code base and clients.

[iloveeclipse]

If the main issue here are duplicated search matches, easiest path would be to add some new global search preference and consistently use it in classic/quick search and related client (JDT, PDE etc) code.

[mickaelistria]

I think it is not realistic

Maybe not. But it's more realistic to not change core because of JDT limitations than the other way round. However, some preliminary steps might be doable, like allowing JDT to work without a dedicated project/nature, and allowing to access classpath & settings for a given IContainer (IFolder or IProject). One possibility can be to have IJavaProject not mapped to a IProject, but to an IContainer.

If the main issue here are duplicated search matches, easiest path would be to add some new global search preference and consistently use it in classic/quick search and related client (JDT, PDE etc) code.

Bad consequences of duplciated resources are everywhere, not only in search. We already have workarounds for it in the Open Resource dialog, but this is really too weak and requires similar hacks everywhere. It doesn't scale. I would really not recommend anyone to follow that track further. I think there is much more hope and correctness and value in improving JDT than on stacking workarounds in Platform.

[iloveeclipse]

JDT has zero resources for that work. So unless you have concrete resources allocated this will stay a plan.

Regarding "hacks" - I don't think it is a hack to filter duplicated search matches from hierarchical projects. It is same kind of a "hack" as hierarchical projects "hack". So if one is considered good, other is not a hack, and vice versa.

[mickaelistria]

JDT has zero resources for that work.

You can't predict the future and you're already wrong about the present. For example, my employer does spend a significant amount of resources on JDT (and JDT-LS), and those people don't count for 0. I'm hopeful JDT will eventually improve on that aspect, although I concede it's not certain nor soon given the difficulty of the task.

I don't think it is a hack to filter duplicated search matches from hierarchical projects.

My experience, as I implemented it twice already, 1 for Project Explorer and 1 for Open Resources, is that those are real hacks to workaround a fundamental problem caused by JDT design that mandates resource duplication while some alternative could be implemented. Such similar hacks would still have to implemented in a dozen of places in Platform and require user buy-in/understanding/preferences/opt-in actions to work well. This is a wrong path; I even regret having spent time to try to hide the ugly truth is some places as it keeps surfacing in many other places; adding inconsistency to the stack of problems caused by duplicate resources.

If you have the opportunity to try other IDEs, you'll realize that they don't have this issue and it's actually making a lot of workflows much better. Both intellij and vscode are open source, you can see in the code that they're not implementing tons of hacks to deal with duplication, they simply don't duplicate. And that's one recurring argument that comes back when you ask people why they don't like Eclipse IDE. On the other hand, in Wild Web Developer, or Corrosion, or others; we do have module-specific assistance without need to duplicate resources either. So it shows it's possible to produce decent tool without duplicated resources.

[iloveeclipse]

Not sure why you always talks about JDT here if the original issue about duplicated search matches, which is not JDT related in any sense?

[mickaelistria]

Not sure why you always talks about JDT here

JDT is the main reason why users have to deal with nested folders (and thus duplicated resources). Make JDT not require an IProject but instead allow and IFolder to get an IJavaProject and then you can work with multiple Java projects and 1 single IProject, no more nested projects, no more duplicated resources, everything is simpler.

[laeubi]

I also think we can't blame JDT here. Saying JDT is the reason for nested folders is simply wrong as you can reproduce the issue with plain projects as well (e.g. when importing maven module builds that don't contain java code and thus has no JDT nature at all). Or if I have PHP projects, JS related stuff, "simple" projects because I like to have more meaningful names that the folder name and so on.

no more nested projects, no more duplicated resources, everything is simpler.

I don't think that's really 'simpler' or will solve this, there is always a world beyond java.

My experience, as I implemented it twice already, 1 for Project Explorer and 1 for Open Resources, is that those are real hacks to workaround a fundamental problem caused by JDT design

I think the fundamental flaw is that there seems to be no way in core(!) to tell that two IResources actually map to the same physical file and that's what people actually complain about, so I think we could add:

• IResource#mapsTo(IResource other) that tell if 'other' actually maps to the "same" as this
• IResource#isMoreSpecific(IResource other) that tells if 'other' is more specific (e.g. has a shorter IPath than this)
• IResource IResource#getMostSpecific() that returns the most specific one or this resource

That way a search could simply filter out stuff conveniently, filter a folder marked as derived even in nested structure ...

[merks]

Note that you can use resource filters to deal with issues like this. For example, for this repository:

https://github.com/eclipse-cbi/p2repo-aggregator

I wanted the root project to be in the workspace so I can edit the files in it in the workspace. But this project of course has "nested" projects. So the working tree looks like this:

But, it looks like this in the workspace:

So I do not repeat the content of each subproject under the folder in the root project.

I do that by filtering the resources like this:

I didn't even need to list what's filtered but rather filter every folder that doesn't start with ..

So in other words, if you don't want to see certain content in the workspace, I filter it out. E.g., I often want to filter out all the target folders created by a local Tycho build because it too results in all kinds of "duplicate" matches when I do a search.

Note that we already have org.eclipse.core.resources.IWorkspaceRoot.findFilesForLocationURI(URI) for finding all resources with a given URI so we don't need other methods directly in the API. We don't know which IFile one is more specific semantically, but we can certainly order/filter by workspace path length. So we don't need to add methods to the resource API to implement utilities.

I personally would just close this issue because it doesn't ask for anything specific but rather makes an observation of facts; facts that reflect the design intent of the resources API. Yes, the same underlying file/folder can appear as more than one IFile/IContainer. That's not a bug that is the design intent. If you don't want that, filter it. If you want it (duplicates) but you want search to behave differently, that's not something to report against the resources APIs. That would be an option for the search engine like ignored Derived resources perhaps it could implement Deeply nested duplicates.

[laeubi]

So in other words, if you don't want to see certain content in the workspace, I filter it out.

We can't expect users of the IDE to be experts of the filter API, so this might work for you but most users will be confused and frustrated and blame it on "Eclipse", so the solution can not be that each users has to filter stuff itself but we should provide the defaults that are most likely match users expectations.

In a setup as you described, I simply would not expect that

• a search of an kind shows duplicate
• I get content from "target" folders
• Editors complaining that a resource is not at a java project and I can't edit them
• ... might be continued ...

If we can solve this with filters, this would be great and we probably just need to add more (default) filters to the IDE...

[merks]

Note that you mention search. As I suggested, that's a place to add an option, but not something to be fixed in the resources API.

A Maven target folder should be marked derived, but it's created in the underlying file system which doesn't know how to mark it that way, though people already figure out how to add it to .gitignore and they are expected to be experts in the filtering "API" that it provides. I'm not even sure where the information about derived is stored...

The editor doesn't complain does it? It's just not as functional as it should be. But it could offer to open the "duplicate" that's actually in a Java project. Or could offer an option to do that automatically. But this is a JDT issue not a resources API issue.

In any case, all the problems must be solved somewhere (else). The resources API is as it is and is working as designed. We can talk and talk (and talk), but if you want to go back to the beginning and redesign it, and redesign JDT on top of that, then you can fork Eclipse as a whole and start with no downstream community from scratch.

Or we could focus on problems we can actually solve within the bounds of the framework....

[laeubi]

Note that you mention search. As I suggested, that's a place to add an option, but not something to be fixed in the resources API.

This was an example and I only suggested adding API because @mickaelistria complained that "workarounds" are implemented multiple times, so it seems we are not offering plugin authors with the necessary tools (or they don't know about because it is hard to find).

though people already figure out how to add it to .gitignore

That's a whole different story, as it is related to versioning files, beside that, (J)git do not has any problems with 'nested' projects and applying the ignore, so if that is setup once by a "git expert" the whole team and every contributor of that project will benefit from that.

I'm not even sure where the information about derived is stored...

it is stored "at the resource" and thus even if at the "project" level the folder is marked, if you import the parent folder as another project it is there not marked, I already suggested to support more persistent storage in Bug 577639 but it did not get much attention.

The resources API is as it is and is working as designed.

Sometimes something is designed and then time passes by and one sees that not all issues where addressed at the initial design phase ... so maybe some kind of a "hierarchy" support could be built in e.g. having a getParentResource() that simply would not use the Workspace root but returning e.g. the folder of a contained project.

redesign it, and redesign JDT

I never stated that I want to redesign it, but all request to improve are always end in "the problems must be solved somewhere (else)" or blamed as "hacks" and thus we are suffering from a solution since 2008, that's 14 years and counting on...

fork Eclipse as a whole and start with no downstream community from scratch

There are other IDEs that even started from scratch in the meanwhile...

Or we could focus on problems we can actually solve within the bounds of the framework....

I'm all for that, so what is the solution? Ask people to stop complaining and redirect them to an eclipse filtering tutorial?

[merks]

Why can't one Eclipse expert add the desired filters for the whole team to enjoy as she does already for the .gitignore? My guess is that 99.9% of the people don't even know this is possible. Strangely, in this discussion it's simply been dismissed out of hand almost as if the concept is ridiculous because people would have to learn something new.

[laeubi]

My guess is that 99.9% of the people don't even know this is possible

I think you found the answer to your question "Why can't one Eclipse expert add the desired filters for the whole team", I think one expert can't work for so many teams at once ;-)

In contrast, .gitignore is well known, simple and even has tooling support (just rightclick and choose Team > Ignore), and is tooling independent (so probably more than one expert available).

as if the concept is ridiculous because people would have to learn something new

The problem is not that people don't want learn something new, and I won't dismiss the idea, it is just far from what usual user expect e.g. configure their IDE right after project import to get some "obvious" things fixed, just keep in mind that the usual users is familiar with file-system-trees but most probably don't know or understand the "resources" concept of eclipse and why the same file is displayed twice (even though it is not the same resource)...

[merks]

It seems to me that something is creating all the .project files; no user is doing that manually. And that something could be smarter about how to import nested projects when the parent folder is also imported as a project, and it could set up such filters automatically. (But then someone is likely to complain that the nested content isn't displayed and they don't understand why.) Maybe the filter could even be easier to specify; a filter to ignore the contents of any folder if it contains a .project file...

[laeubi]

It seems to me that something is creating all the .project files; no user is doing that manually. And that something could be smarter about how to import nested projects when the parent folder is also imported as a project

I sometimes just copy project files or I check them out, or ... so I think creation time is not correct.

a filter to ignore the contents of any folder if it contains a .project file...

This would imply any project is imported in the workspace... that's not necessarily the case.

[merks]

I'm out of suggestions given that every suggestion is clearly fatally flawed for one reason or another...

[laeubi]

every suggestion is clearly fatally flawed for one reason or another

I don't think it is "fatally flawed" just more cases beside the simple ones :-)

But given that the hierarchy view of project explorer works quite flawless, I'm really curious why the same technique can't be applied to other parts of the code... e.g. Bug 573535 is a good example and close scoped, so the question is:

What would be needed for the Problems view to understand the hierarchy without require to implement a custom solution for this particular view...

If that is solved, other parts of the code could use the same (generic) technique to understand the hierarchy of projects and then even your suggested filter would work by having a "Hide nested resources" that checks for a .project file and if the project exits in workspace and is a child of the current project.....

[mickaelistria]

I also think we can't blame JDT here. Saying JDT is the reason for nested folders is simply wrong as you can reproduce the issue with plain projects as well (e.g. when importing maven module builds that don't contain java code and thus has no JDT nature at all)

You can blame any technology that makes it necessary to have nested projects. Nested projects are not well supported by the workbench, it's a fact that was true for the last 21 years and will remain true forever.

I don't think that's really 'simpler' or will solve this, there is always a world beyond java.

This world care way less about nested folders. People who are repeatedly complaining about the issues of duplicated resources are Maven or Gradle users at the moment who are forced to deal with nested projects because of JDT.

I think the fundamental flaw is that there seems to be no way in core(!) to tell that two IResources actually map to the same physical file and that's what people actually complain about

Wrong. One can compareresource.getLocation(). The fundamental flaw is that there are 2 distinct resources for the same underlying locations.

Note that you can use resource filters to deal with issues like this

That also leads to incomplete results in several locations because the IResourceVisitor does not visit the excluded resources. So the children are really not visible in many places. Eg a Search with "Selected resource in Project Explorer" won't show the occurence in filtered resources and will give an impression there is no match at all while there may be some in nested projects. Also, when a resource is filtered and the children problem are not imported, the resource simply seems totally missing and the project not usable at all, even less than without filter. It's how EGit source tree is configured, and it's not really convenient for many cases: it forces you to think import all projects at once, instead of just allowing to import the root tree + 1 or 2 subprojects. In term of usability, as it's easy for end-users to misuse it, it leads to a worse user experience: no resourece when 1 is expected is less capable than 2 duplicate resources. Filtered Resources are not fully viable either because they introduce new problems

As I suggested, that's a place to add an option, but not something to be fixed in the resources API.

An option is yet another workaround (occurence number 4 maybe), but I agree the resources API needs no fix. Nested projects suck by design of the resources API and there is no forseeable way to fix it. However, there seems to be some possibility to get language support working without nested projects...

This was an example and I only suggested adding API because @mickaelistria complained that "workarounds" are implemented multiple times, so it seems we are not offering plugin authors with the necessary tools (or they don't know about because it is hard to find).

Workarounds is that the similar code to implement filtering is to be implemented in every IResourceVisitor, which really doesn't scale.

So filters don't work well, ssuming we can change all resource visitors don't work well while the necessary API is there. The clear issue is that Eclipse Platform doesn't support well nested projects, never have and -IMO- never will. Now it's up to downstream to deal with that fact and consider improving their design if they want to make their user happier.

I think we can close it here. Work is better placed downstream.

[laeubi]

@mickaelistria I really don't get the

Nested projects suck by design

for me they always have worked well, and if the really "suck by design" the projects itself are flawed and we should just drop the need to have them at all, then we also don't need workspaces and so on ...

I think maven is a perfect example where such nested projects would be needed, as modules could be referenced not only to direct childs, there is no other way to represent them in a consistent way, but maybe maven modules also "suck by design"??

[mickaelistria]

for me they always have worked well,

Contextual search shows wrong resources, resources are duplicated in several views and dialogs, some operations are performed multiple times, workaround are implemented here and there to mitigate the duplication.... It makes user experience incomplete (please read twitter or other channels where users bash Eclipse IDE, duplicated resources a recurring issue) and requires significant effort to adopters if they want to present things more nicely or in a more optimized way to users. Now I'm counting 4 occurrences where I've built workaround for this issue: Project Explorer, Open Resources, File Search optimization, Quick Accesss... Having to implement similar things 4 times for the same reason is a hint that "it sucks by design". A better design wouldn't require such adoption effort to provide a good user experience or good performance. And this issue is a fundamental issue of duplicated resources which is a natural consequence of nested projects.

the projects itself are flawed and we should just drop the need to have them at all, then we also don't need workspaces and so on

Exactly. The language logic must do their best to work without putting requirement on the workspace or resource model that surface as bad user experience. Other IDEs got it better, LSP got it better.

I think maven is a perfect example where such nested projects would be needed

I don't think so. No need to have a IProject to get an IMavenProjectFacade... This is the topic of https://github.com/eclipse-m2e/m2e-core/issues/682

[cobexer]

I also think we can't blame JDT here. Saying JDT is the reason for nested folders is simply wrong as you can reproduce the issue with plain projects as well (e.g. when importing maven module builds that don't contain java code and thus has no JDT nature at all)

You can blame any technology that makes it necessary to have nested projects. Nested projects are not well supported by the workbench, it's a fact that was true for the last 21 years and will remain true forever.

I don't think any Integrated Development Environment is in the position to demand the world to change, either an IDE supports current practices of the world or the world has to move on to another IDE.

Threating "eclipse.platform.resources" APIs as finished and set in stone doesn't sound like the best approach to me.

Now I'm counting 4 occurrences where I've built workaround for this issue: Project Explorer, Open Resources, File Search optimization, Quick Accesss... Having to implement similar things 4 times for the same reason is a hint that "it sucks by design". A better design wouldn't require such adoption effort to provide a good user experience or good performance. And this issue is a fundamental issue of duplicated resources which is a natural consequence of nested projects.

Requiring a workaround in every component that interacts with resources could also mean that different behavior of the platform would improve the overall developer and user experience - possibly with different tradeoffs.

You identified a number of usecases that would change behavior (maybe percieved as breaking) if nested resources were removed in core and there exists a lot of evidence of usecases currently perceived as broken.

Eg a Search with "Selected resource in Project Explorer" won't show the occurence in filtered resources and will give an impression there is no match at all while there may be some in nested projects. Also, when a resource is filtered and the children problem are not imported, the resource simply seems totally missing and the project not usable at all, even less than without filter.

So if IResourceVisitor would not cross (imported) project boundaries by default (and might have a flag telling it to do so) and resources were to be deduplicated (parented to their nearest project)

would this create or solve more problems?

[mickaelistria]

So if IResourceVisitor would not cross (imported) project boundaries by default (and might have a flag telling it to do so) and resources were to be deduplicated (parented to their nearest project) would this create or solve more problems?

It would solve duplication problems and simultaneously it would create problems that some "visits" would miss some expected results.

[iloveeclipse]

Looks that constructive proposals are not welcome here, but I will try last time.

We have multiple constraints here:

• we can't break API
• we can't rework resources model to get rid of project concept
• we can't rework resources model to get rid of nested projects
• we can't rework resources model to get rid of linked resources concept
• we have no resources to fully rewrite JDT/PDE to support "classpath per folder" (we don't have resources to even properly review incoming PR's, not saying anything about existing compiler bugs etc).
• even if we would rewrite JDT/PDE, it will break all existing clients
• even if we would rewrite JDT/PDE, it will not fix issues for NON-Java nested projects and linked and virtual resources

And we have a very simple user story:

• no duplicated matches for "same" resources in search / quick search
• no duplicated markers for "same" resources
• no different editors for "same" resources
• eventually filtered resources in explorers if they are shown somewhere else already

With that, even if it sounds like a boring work and some name that boring work a "hack", why not spend time to look at the requirements above and think "what kind of API we may need to add" to implement at least these small wishes? Not re-inventing entire workspace concept or JDT core model, but evolve it and adopt to requirements above.

The common thing there is that in some places we only want see resources with "longest paths" and don't care about resources with "shorter" paths. So while adding some element to the search results or tree or markers etc we might want to know

• if that resource is "short alias" for some other one
• or we want know the "longest path" alias for given resource

With that information above, provides as new IResource API one could start do boring work and provide "hacks" to make users happy. I see there is something that seem to be aware about "duplicated" resources (aliases) org.eclipse.core.internal.resources.AliasManager. I don't know if that is also takes care about linked or virtual resources (probably not), but I could imagine that this can be used / extended to compute required information.

So an IResource might get

• an extra (internal) flag "hasAlias" (either as boolean flag or just as a session property), to avoid constant computation
• new API like boolean isAlias()
• new API like List<IResource> getAliases() sorted by path length
• new API like IResource getFirstAlias() returning aliased resource with shortest path
• new API like IResource getLastAlias() returning aliased resource with longest path

... and with that, client might do boring things like:

// filter least interesting (assuming we have only one duplicate) :
if (resource.isAlias() && resource.getFirstAlias() == resource) return true;

// replace with most interesting :
if (resource.isAlias() && resource.getLastAlias() == resource) return resource

etc...

The main problem I see is that to compute this information we will need time & memory, so the real difficulty is not to provide that API but to make sure it does scale and does not block, also for workspaces with > 100.000 of resources and lot of duplicates.

[Bananeweizen]

Have you checked whether the available filters in the project explorer would be helpful? Be aware those are not available in the package explorer unfortunately.

[mickaelistria]

Have you checked whether the available filters in the project explorer would be helpful?

This is only UI of 1 view, not applied to the overall workspace model nor to all UI, so even when using UI Filters, it's still very easy to get into a state when the "wrong" resource is actually used (eg using File Search on parent folder, or some other UI components that do not de-duplicate resources).

[cobexer]

Working code to test Eclipse with the changed behavior: https://github.com/eclipse-platform/eclipse.platform.resources/pull/169

A piece is still missing: Triggering a refresh of the parent of a folder that is imported as a project / maybe generally after opening / creating a project. BUT: I'm pretty sure changing filters has the same glitches (or has code to trigger a refresh - I didn't have time to check this yet)