make large numbers of resources in sidebar more manageable

nicks commented 4 years ago

When one has a lot of resources, the sidebar is noisy and also might require scrolling to see that a resource is unhealthy. Local resources enable use cases that lead to many more resources in the sidebar.

Some spitballing:

Hide all healthy resources under a "healthy resources" group.
would probably need some extra control like a "pin" button or tiltfile directive for resources you want to see all the time
Allow users to define resource groups that can be collapsed/expanded in the sidebar

nicks commented 4 years ago

I've also thought about tying this into Tilt Cloud somehow - i.e., you can re-arrange the services dynamically, and Tilt Cloud "saves" your preferences (similar to how most browsers do Bookmark sync)

vkorbes commented 4 years ago

YES PLEASE

crhayes commented 4 years ago

Resource groups would be 💯💯💯.

We have a bunch of micro services, and one use case that would be nice is to expose local_resources for migrating/rolling back, seeding/unseeding each service. That would quickly become a nuisance with multiple dozens of resources in the sidebar.

The ability to create folders like "Migrations" or "Seeders" would be sweet.

talarari commented 4 years ago

Also a simple text filter on top of the sidebar would go a long way.

nicks commented 4 years ago

One good example of this problem is the Gloo installer in this Tiltfile:

https://github.com/tilt-dev/tilt-extensions/blob/master/helm_remote/test/Tiltfile#L10

you really want some better way to group these different resources together. @KOGI suggested something like

resource_group('gloo-group', ['gloo', 'gloo-discovery', 'gloo-gateway', 'gloo-gateway-proxy', 'gloo-gateway-certgen', 'gloo-crds-install', 'gloo-crds-ready'])

so that you could use resource_deps to depend on the whole bundle of resources

kevin-lindsay-1 commented 4 years ago

Per #3587, I'm pasting my previous comment here:

Additionally, it would be really nice if Tilt was capable of grouping these namespaced resources instead of putting them all in the same line. e.g.:

resources:
  - install deps (local)
  - build (local)
  - k8s_service (docker_build/k8s_resource)
  - serviceA:
       resources:
         - install deps (local)
         - build (local)
         - k8s_service (docker_build/k8s_resource)
  - serviceB:
       resources:
         - install deps (local)
         - build (local)
         - k8s_service (docker_build/k8s_resource)

So that you could collapse things such as:

resources:
  - install deps (local)
  - build (local)
  - k8s_service (docker_build/k8s_resource)
  - serviceA
  - serviceB

And while I'm at it, I'm pasting my comment from #3530 here as well, because it seems very related.

While that's being thought through, I'd like to [possibly tangentially] raise up a consideration for showing deps as well, possibly by inferring a rough order from which resources have dependencies.

I recognize that the current view might not be ideal for such a request, because you're basically trying to represent a Directed Acyclic Graph via one-dimensional set of points. My gut says a design accomodation could be to create collapsible groups for referenced Tiltfiles and possibly incorporate a light border between dependencies (A and B require C, order would be C ; A, B, and more complex orderings could be inferred from the dependency graph, allowing manual override for the picky).

Garden's attempt to make the dependency graph a literal graph is nice, but IMO it uses up a lot of screen real estate where you could introduce a visual design pattern to convey similar meaning. For example, perhaps if you highlight a particular resource, you could have two different highlighted colors on the other resources; one color for the selected resource's dependencies, another color for resources that depend on said resource. This combined with an inferred order/grouping on dependencies would make it very clear when in the deployment process things broke.

In summary, my design proposals for consideration:

Infer an order for dependencies
- Top-down or bottom-up by default (Tilt's preference); overridable
- Allow manual specification of order
Allow collapsible grouping of resources
- Referenced Tiltfiles would be grouped by default
When selecting a resource (s), dependencies are highlighted
- Children c of s are highlighted a specific color
- Parents p of s are highlighted a different color

kevin-lindsay-1 commented 4 years ago

To add to the discussion, I think a way to handle this in a backwards-compatible way would be to add a new parameter to include a la:

/Tiltfile

include('./services/vehicle/Tiltfile', group='vehicle')
...
k8s_resource('service')

/services/vehicle/Tiltfile

include('../train/Tiltfile', group='train')
include('../plane/Tiltfile', group='plane')
include('../automobile/Tiltfile', group='automobile')
...
k8s_resource('service')

Other Tiltfiles

/services/train/Tiltfile

/services/plane/Tiltfile

/services/automobile/Tiltfile

...
k8s_resource('service')

Resources created

Flat representation

service
vehicle-service
vehicle-train-service
vehicle-plane-service
vehicle-automobile-service

Grouped representation

service
vehicle
- service
- train
- service
- plane
- service
- automobile
- service

maiamcc commented 4 years ago

Ooh, I like that proposed syntax. Thanks for some great and well fleshed-out ideas @kevin-lindsay-1 !

bobjackman commented 4 years ago

A quick and dirty photoshop of what the UI could look like once this is implemented (where each group can be expanded/collapsed via the little arrow icon):

jordykirkman commented 4 years ago

I'm going to have a go at a first pass for really basic resource grouping using @KOGI 's suggested resource "group" syntax as a start. Some details I'm thinking for the UI:

Warning/error color indicator on any resource under a group will change the color of the top level group
Show number of resources ready out of total next to group name
Default groups as collapsed

bobjackman commented 4 years ago

@jordykirkman possible top-level indicators?

kevin-lindsay-1 commented 4 years ago

IMO single color is all I need to know there's an issue, and mixing colors like that sounds like an accessibility concern

hyu commented 4 years ago

Thanks for creating that mock, @KOGI! Since this proposed change affects the interface, it's great to think through the visual details. So I really appreciate you kicking off that discussion. Some quick feedback:

In this mock we're using indentation to show hierarchy, and keeping the appearance of each resource item basically the same. But there's another strategy I'd be interested to explore:

Group header is visually distinct from its child items.
When expanded, each group looks clearly sectioned off from other sections. (e.g., through vertical spacing or some background or border that unites the group.)

This could address some of the other potential problems in the design:

You have the expand/collapse trigger inside the status indicator, but that that's where we display a number indicating the number of alerts. (It impacts accessibility if we solely rely on color to indicate status.) Could we expand/collapse groups another way, perhaps by just clicking anywhere on the group header? In the current design, clicking on a resource item shows the respective logs / alerts / etc in the main pane. But do we expect that clicking on a group will show the multiplexed logs for all its associated resources? (Sounds cool, but I'm genuinely not sure how useful that is.)
If we're not using indentation to indicate hierarchy, it helps to keep the alignment of data like the resource name, timestamp, trigger button, etc. That makes it easier to quickly scan down the page and spot things like, if a particular timeago is longer than you expected.
I like the idea of being able to quickly see the resources running of the total resources in a group. But when I see that "3/3", I don't immediately understand that it refers to "running/total resources". There may be another treatment that makes that more obvious.

(CCing @smombartz — who also did a lot of work on the most recent design of the sidebar. In case he also wants to weigh in!)

smombartz commented 4 years ago

thanks everyone! 🙏❤️ this is a great idea! I've made note, we'll consider everyones input and report back with a design to get your feedback once we have it.

bobjackman commented 4 years ago

Might also be useful for all groups to be collapsed by default, unless there's a failure/alert within the group. Then that group header is highlighted with the appropriate color, and the group is automatically expanded so the items can be readily seen.

smombartz commented 4 years ago

Hey @KOGI, @kevin-lindsay-1, @jordykirkman, @crhayes and @talarari! thanks again for your feedback – we created this initial design mockup taking all your input into account. what do you think? one note: 2nd level grouping, i.e. a group within a group is out of scope for the time being.

kevin-lindsay-1 commented 4 years ago

@smombartz I like it!

Nitpicks:

The number of errors (e.g. the 2 next to AUTHORIZATION) may end up having strange placement, due to the fact that group names can vary in size. In most design systems, group names as a design "surface area" aren't typically where I'd expect to look for parse-able output.
- Suggestions:
- Perhaps move the number of errors into the same design space as the <up>/<total> indicator, perhaps [<errors>/]<up>/total>, meaning when there are 0 errors on a group of 5, where 3 services are currently up, it would display as 0/3/5 or 3/5. Once 4 are up and 1 errors, it would show 1/4/5

Future considerations:

If the group GLOO has group VAULT as a dependency, how do we convey that? Specifically, how do we convey the mental "line" of it requiring it, as well as how do we display the vertical ordering of elements given that dependency? A hover highlight or temporary set of lines might work, or maybe a new color added to the palette.

maiamcc commented 4 years ago

Hey folks! Given the number of considerations with the UI, I'd suggest that folks here interested in writing code focus on getting the information TO the frontend, and the Tilt design/frontend folks take it from there (though knowing them, they will likely reach out for feedback along the way).

In particular, I suggest @hyu specifies the preferred format for this information to show up in the frontend state[1] -- then you all can work on piping these settings through from the Tiltfile -> engine state -> frontend state.

My only other request is that a Tilter signs off on the proposed Tiltfile syntax (did y'all want to use KOGI's suggestion above? Or something else?[2])--after that, you can start work on a PR/series of PRs. (The team prefers smaller PRs because it makes reviews, rollbacks etc. easier: feel free to reach out to one of us if you want help figuring out how to break this work into smaller PRs.)

1: is view.resources now view.groups where each group is a []resources? is there a separate field view.groups field that maps groupName -> []resourceNames? (I'd assume the latter for ease of implementation/backwards compatibility but I don't presume to know how to frontend.) 2: FWIW @kevin-lindsay-1 your suggestion for grouping from include statements is intriguing but I'd propose that that should wait until we have basic grouping implemented.

bobjackman commented 4 years ago

:heart: all of this!

@smombartz just a couple thoughts:

What would it look like if there were both groups and individual (non-grouped) resources at the top-level? Or will that even be possible? (I feel like it should be -- groups should be completely optional)
Nitpick: I can tell that the filled/unfilled circle at the left of the group name is being used to indicate whether the list is currently expanded or not, but I also feel like it's a little confusing -- an empty circle suggests some sort of incompleteness that makes me wonder if there might be something wrong. Considering we can pretty obviously see when a group has been expanded, I'm not sure the filled/unfilled circle distinction is really adding anything.

@kevin-lindsay-1 re: future considrations: showing dependency links: AFAIK any sort of dependencies between any resources aren't currently being shown at all right now, so, pedantically speaking, this new group functionality shouldn't cause any problems. (FWIW, Garden shows a full dependency tree, which is really cool and would be awesome for Tilt!)

jordykirkman commented 4 years ago

@smombartz I really like that ui! Though I agree that the empty circle when expanded is a little confusing. It initially read to me like something is still spinning up. What will be in that area while services are starting?

For the tiltfile syntax I like KOGI's simple suggestion

kevin-lindsay-1 commented 4 years ago

@maiamcc

is view.resources now view.groups where each group is a []resources? is there a separate field view.groups field that maps groupName -> []resourceNames?

It sounds to me like given a single tiltfile, you should be able to have a type definition along the following:

// TypeScript flavor

interface ResourceHaver { // lol excuse the name
  type: 'tiltfile' | 'group' | 'resource'; // for the purpose of discrimination at runtime
  groups: Group[];
  resources: Resource[];
}

interface Tiltfile extends ResourceHaver {
  groups: Group[];
  resources: Resource[];
}

interface Group extends ResourceHaver {
  name: string;
  groups: Group[]; // note the nested groups
  resources: Resource[];
}

interface Resource {
  name: string;
  type: ResourceType; // enum
  ...
}

My only other request is that a Tilter signs off on the proposed Tiltfile syntax

At the moment, I see 2 short-term use cases and 1 long-term use case that the syntax needs to solve for:

1. Local tiltfile groups, with support for nested groups in future

[resource_]group('group_name', ['resource_name', ... ]) // @KOGI fine with either `resource_group` or `group`
group('parent_group_name', ['resource_name', 'group_name', ...])

// parent_group_name.group_name.resource_name

If discrimination is more difficult to implement, harder to read, or more computationally expensive, you could separate them out into different parameters:

group('parent_group_name', resources=['resource_name'], groups=['group_name'])

2. Place remote tiltfile in group

include('./services/A/Tiltfile', group='A')

// A.parent_group_name.group_name.resource_name

3. (Long term) Override remote tiltfile groups (in the case of a shared dependency or a different slice of the architecture)

// Simple example
include('./services/A/Tiltfile', group='A')

move(from=['A', 'parent_group_name', 'group_name'], to=['a_new_group_name'])
// A.parent_group_name.group_name.resource_name => a_new_group_name.resource_name

---
// More complex example

include('./services/A/Tiltfile', group='A') // A.service
include('./services/B/Tiltfile', group='B') // B.service

rename_resource(from=['A', 'service'], to='A') // A.A
rename_resource(from=['B', 'service'], to='B') // B.B

group('services', [])

move(from=['A'], to=['services']) // services.A
move(from=['B'], to=['services']) // services.B

your suggestion for grouping from include statements is intriguing but I'd propose that that should wait until we have basic grouping implemented.

Agreed; syntax should be able to come intuitively from a schema.

@KOGI

I can tell that the filled/unfilled circle at the left of the group name is being used to indicate whether the list is currently expanded or not, but I also feel like it's a little confusing -- an empty circle suggests some sort of incompleteness that makes me wonder if there might be something wrong. Considering we can pretty obviously see when a group has been expanded, I'm not sure the filled/unfilled circle distinction is really adding anything.

Thought the same thing.

re: future considrations: showing dependency links: AFAIK any sort of dependencies between any resources aren't currently being shown at all right now, so, pedantically speaking, this new group functionality shouldn't cause any problems. (FWIW, Garden shows a full dependency tree, which is really cool and would be awesome for Tilt!)

Agreed; once it's time for dependencies then the existing schema should simply be able to be updated for the sake of advanced functionality under the hood (assuming they're not already implemented to some extent), hopefully making the UI considerations more of a design problem rather than an implementation problem.

Honestly, it's a fairly tough challenge to try to represent what is effectively a DAG in "linear" interpretation, but I don't think I'd want to go the full Garden approach unless necessary, because a 2D graph in that fashion would likely be a lot of work, and use a lot of screen space when it's present, meaning it would likely interrupt flow, where normally the ability to casually glance at the bar of resources is enough to get the picture. If there's a way to display the dependencies in such a way where they don't take over the view, I'm all for it. If displaying them would require to drastically inflate the surface area of the resources bar, then I'd probably not use it that often.

That said, if someone uses Tilt in production, a graph could be a very nice way to view the services on a TV or something, but honestly that's a pretty edge use case. Really easy feature to sell to business people, though, if it's eye candy.

hyu commented 4 years ago

@KOGI — in the mock, Tiltfile looks the same as non-grouped resources would look. I think they can coexist nicely along groups! But it does leave the question of whether we show groups all pinned to the top, or interspersed among other resources based on order of appearance in your Tiltfile.

I also agree with the feedback from @KOGI , @jordykirkman , and @kevin-lindsay-1 about the filled/unfilled circle. It does seem like it's group resource status, rather than expand/collapse! So, to reduce ambiguity, I agree that we should keep that circle consistent and just let you see expand/collapse from other obvious visual cues.

Re: @maiamcc's comment about my preferred format. I haven't dug too deeply into this, but just from my work with @smombartz on the mocks and quickly reviewing the discussion above, I'd like a better understanding than I have now of the best way to quickly get the info I want to display in the group header: total errors / warnings / currently running / total sub-resources / timeago from most recent resource. I am lazy and don't want to iterate through each group's child resources to figure it out! It also seems potentially expensive to do all this on the frontend, especially if groups can contain a ton of resources.

bobjackman commented 4 years ago

@hyu I'd be interested to see a mock-up that shows both grouped and non-grouped resources at the top level. It seems to me, though, that the differences between the main group bar and the non-grouped resource bar (one with the dot and dark background, the other with the big status indicator "tab" and a lighter background), might increase the cognitive load when glancing at the list, making it harder to see at a glance if everything's ok or not.

Maybe it would be ok to keep the dark background and the rest of the design, but use the same big "tab" status indicator on the left side instead of the little circle/dot?

maiamcc commented 4 years ago

So many great ideas being kicked around here! Unfortunately Team Tilt doesn't have bandwidth to implement this right now but we'd be excited to help a contributor(s) do it.

I've fleshed out requirements for an MVP of this feature in #3643 -- if you're interested in contributing, comment over there!

For further design discussion of the MVP, see #3644.

maiamcc commented 4 years ago

Ping! I thought I saw some folks upthread who were interested in contributing to this feature? If that's you, drop a comment in #3643

jordykirkman commented 4 years ago

It was looking like we were ready to try to tackle this, but some other priorities arose and I won't have time to start on this immediately. This is still on our radar though. If i get to a point where I can jump onto either #3643 or #3644 I will chime in on on those threads. I believe @KOGI is in a similar spot

maiamcc commented 4 years ago

@jordykirkman @KOGI fair enough! Like I said, we're not likely to get to this internally in the immediate future, but we'll post here if we pick it up--and if ever either of you wants to jump in here, please do! And thanks for all your good thoughts an energy on this topic :)

nicks commented 4 years ago

I liked this idea from Vignesh T.V. in the tilt channel:

Not sure if you offer a filter option for buttons though. Btw, a VSCode like behavior Ctrl+Shift+P or spotlight search would make sense to quickly navigate between various buttons in tilt

tvvignesh commented 4 years ago

@nicks Probably I will stick to Github and continue the conversation rather than you having to spend time copy-pasting stuff 😅

landism commented 3 years ago

This thread has a lot of great ideas! Since it was created, we've shipped:

pinning resources
a simple resource name filter
a button to sort failing resources to the top
and, as of today, resource grouping: https://docs.tilt.dev/tiltfile_concepts.html#resource-groups

ZacxDev commented 3 years ago

I'd like to suggest a fuzzy finder that allows one to quickly focus a tilt resource by pressing a keybinding to open a floating search input and typing in a query that is fuzzy matched against all tilt resource names; clicking/pressing enter on a search list item would then focus the resource the same way clicking on it in the sidebar would.

lizzthabet commented 2 years ago

i want to second @landism's comment that there have been so many great ideas on this issue! with tilt's resource grouping feature, i think the core of this issue has been addressed, so i'm going to close it.

based on @ZacxDev's comment, i've opened up #5716. if there are other specific requests or pain points related to managing large numbers of resources, feel free to open up new issues!

tilt-dev / tilt