search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Richer software environment info #196

Open fjetter opened 2 years ago

fjetter commented 2 years ago

The information retained for a given software environment should be enriched with more information. Specifically required are things like

Example of the currently available info image

ntabris commented 2 years ago

Why/what’s are you trying to do?

This information is retained in the database.

Also, it’s not clear what the future of existing software environment feature will be.

On Thu, Sep 1, 2022 at 4:32 AM Florian Jetter @.***> wrote:

The information retained for a given software environment should be enriched with more information. Specifically required are things like

  • Creation timestamp
  • Update timestamp
  • conda list of actually installed packages
  • User who created it (Optional / nice to have)

Example of the currently available info [image: image] https://user-images.githubusercontent.com/8629629/187893639-efb64dc3-585a-43cf-8ca2-017c9283f016.png

— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/196, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBFDHXVPMDE2Z6CUE532J3V4CA2TANCNFSM6AAAAAAQCG272Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

fjetter commented 2 years ago

Why/what’s are you trying to do?

Well, without additional information these software envs are basically one-offs. Apart from transitive dependencies that may be relevant, I have no idea what coiled version is installed in the above example. Would this software env even work right now?

Therefore, I need the full conda list to see if this software env suits my needs.

Timestamps are useful to sort through the list of available environments, e.g. to filter on the most recent ones or to reject obviously stale environments. (yes, I believe the UI should also be able to filter but that's another thing)

The owner/creator is important when working in a team. For instance, I don't want to update/delete an environment that I am not "owning" without checking in with my colleague first.

ntabris commented 2 years ago

It sounds to me like that more or less boils down to a bunch of things that package sync is intended to address. Is that not how you're seeing this?

fjetter commented 2 years ago

It sounds to me like that more or less boils down to a bunch of things that package sync

For ad-hoc stuff, yes.

If the intention is to remove software environments entirely you can ignore this ticket. If they are there to stay in some kind of way, I think richer information is still relevant

dchudz commented 2 years ago

@fjetter It'd still be really nice to understand more about the use case. What situation are you in when you want this? You've noticed an existing software environment and want to know if it meets your needs?

Happy for you to chat with one of us if that's easier than typing.

(I'm not trying to resist the feature request... it sounds easy enough. But I think it's important for us to try to really understand what problem we're solving for you.)

fjetter commented 2 years ago

When I encountered this I wanted to start a coiled cluster with reasonably new versions (a nightly build would've been perfect but that's not available yet, see https://github.com/coiled/feedback/issues/197)

I didn't want to build a new software environment because it's slow so I was looking to reuse an existing one. Without this additional information, it's very hard to judge if a software environment is OK to be used. This particular one I showed is too old for me (dask versions too old). I could've iterated manually through the entire list but instead I wanted to just sort it by creation/update timestamp to see if there is anything useful in there.

FWIW For most things I'm doing, package_sync is not a great option, either. I often want/need to recreate a cluster and package sync is relatively slow so I typically prefer to use a pre-built software environment instead.

dchudz commented 2 years ago

Again I'm not saying we shouldn't do your idea but:

For what you were doing, the easiest option is building a software environment from an existing docker image. Which is extremely quick!)

(I realize you didn't know that at the time but now you do...)

On Fri, Sep 2, 2022, 8:04 AM Florian Jetter @.***> wrote:

When I encountered this I wanted to start a coiled cluster with reasonably new versions (a nightly build would've been perfect but that's not available yet, see #197 https://github.com/coiled/feedback/issues/197)

I didn't want to build a new software environment because it's slow so I was looking to reuse an existing one. Without this additional information, it's very hard to judge if a software environment is OK to be used. This particular one I showed is too old for me (dask versions too old). I could've iterated manually through the entire list but instead I wanted to just sort it by creation/update timestamp to see if there is anything useful in there.

FWIW For most things I'm doing, package_sync is not a great option, either. I often want/need to recreate a cluster and package sync is relatively slow so I typically prefer to use a pre-built software environment instead.

— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/196#issuecomment-1235419050, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJKQRXDZ6GSEFLXXIKQMYLV4HUMFANCNFSM6AAAAAAQCG272Y . You are receiving this because you commented.Message ID: @.***>

fjetter commented 2 years ago

(I realize you didn't know that at the time but now you do...)

I did but I don't have a docker image

ntabris commented 2 years ago

I did but I don't have a docker image

And that's a separate issue, right? #197

Or does that not cover your use-case?

fjetter commented 2 years ago

I think this request is still valid and reasonable if we continue to provide software environment builds. Just because my specific situation could've been handled differently doesn't mean this is not worth doing.

If we keep coiled.list_software_environments we should extend the information provided.

FabioRosado commented 2 years ago

Just to throw a wrench into the discussion, when we call the backend for the list_software_environments we get a bunch of information that we aren't showing to the users (such as when the latest build was created). Can't remember exactly the data structure, but I know that we are excluding a bunch of keys from the payload before returning the dict of software envs.

dchudz commented 2 years ago

Yep, agree the request is valid.

We just like to understand users' problems as fully as is feasible. I think that's helpful regardless of whether we end up impending your suggestion tomorrow or next year.

On Fri, Sep 2, 2022, 8:56 AM Florian Jetter @.***> wrote:

I think this request is still valid and reasonable if we continue to provide software environment builds. Just because my specific situation could've been handled differently doesn't mean this is not worth doing.

If we keep coiled.list_software_environments we should extend the information provided.

— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/196#issuecomment-1235473144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJKQRUYKZ6ZHOSK6ZSXRTDV4H2OVANCNFSM6AAAAAAQCG272Y . You are receiving this because you commented.Message ID: @.***>