build should continue if resource cache is not available

[chrishiestand]

It seems reasonable that, by default, unavailable cache should not stop a build. Unfortunately the build hangs because the cached image is not available and concourse does not know it is an optional resource. This is operationally burdensome because it means the first time you run some pipelines you must setup the configuration without cache, build and push, then modify the pipeline to add cache.

Or I may be misunderstanding the instructions?:

cache_from: Optional. An array of images to consider as cache, in order to reuse build steps from a previous build. The array elements are paths to directories generated by a get step with save: true.

here is my pipeline snippet:

- name: image-test
  type: docker-image
  source:
    repository: my-repo/test

jobs:

- name: build-push-deploy

  plan:

  - aggregate:
    - get: git-test
      trigger: true
    - get: image-test
      params:
        save: true

  - put: image-test
    params:
      build: git-test
      dockerfile: git-test/Dockerfile
      tag: git-test/.git/ref
      tag_as_latest: true
      cache_from: [image-test]
    get_params:
      skip_download: true

[eedwards-sk]

This is like year 3 of me trying to use concourse and this issue still hasn't been solved? Seriously?!

What is the point of a cache if it fails when it doesn't exist? That makes zero sense.

[eedwards-sk]

Is this actually solved?

Using OP's example...

- name: build-push-deploy

  plan:

  - aggregate:
    - get: git-test
      trigger: true
    - get: image-test
      params:
        save: true

  - put: image-test
    params:
      build: git-test
      dockerfile: git-test/Dockerfile
      tag: git-test/.git/ref
      tag_as_latest: true
      cache_from: [image-test]
    get_params:
      skip_download: true

If image-test doesn't exist yet, it's still going to hang at "waiting for suitable set of input versions", which is what this issue was opened for, unless I'm misunderstanding something.

[vito]

@eedwards-sk Looks like the PR that 'resolved' this was only for the cache: true case, not cache_from:, so it shouldn't have been marked as fixing this issue.

I don't think much can be done for this issue, though, at least in the resource's implementation. There is no such thing as optional get steps in Concourse and no one has proposed or PR'd such a thing. That would be a core Concourse feature, outside of the scope of any one resource type, so I'm not sure it makes sense to have an issue open here anyway.

Caching is already captured as a sore point in https://github.com/concourse/docker-image-resource/issues/190 which has led to a lot of progress outside of the scope of this resource type, which will likely be retired some time soon. Progress is currently being made in the new registry-image resource and builder task. Once more progress is made on concourse/rfcs#7 to improve the UX of the builder task we can start to deprecate this resource type.

The builder task is currently experimental but already supports caching by way of task caches, and doesn't have the problem described in this issue.

[vito]

This is like year 3 of me trying to use concourse and this issue still hasn't been solved? Seriously?!

If nothing has been done on an issue, that means it needs someone to help out. Perhaps yourself.

As someone who cares about the project and knows how much work there is to do, it's really not fun to see "this hasn't been done yet?!?!?!". It's dismissive of the work that we do day in and day out. It's condescending, it's demotivating, and it's extremely unhelpful. There is no magical team that can execute on every single GitHub issue. If it hasn't been done in 3 years, it's because you haven't done it either.

Be the change that you want to see in the world. Open-source is not slave labor. You get out what you put in.

[eedwards-sk]

@vito

I appreciate the response and context, vito.

I made that comment you're quoting back in October. Apologies if it didn't convey the attitude you would prefer.

I do appreciate the contributions everyone is always making to the project, it's why I like to use open source tools in the first place.

You're right, complaining "why doesn't this work yet?" for something that has barely had a chance to be worked on is not very nice.

Personally though, I don't think that completely invalidates criticism over a lack of features that haven't been prioritized for years.

As time goes on, other tools have begun to "standardize" some expectations around CI solutions, like sidecars or image caching, so it may make these types of issues stand out more, in comparison.

Edit: Also, for posterity, the fact that it IS open source means you're not the target of frustration in "Seriously!?" -- it's frustration over no-one having solved it. I don't expect you or the concourse team to take on every improvement yourselves. To your point, we should try to close these gaps as a community. For myself though, Golang is a bit intimidating. Having contributed in many other languages, it doesn't seem straightforward to jump into Go. However, I'm still hoping to get familiar with Go this year.

[vito]

@eedwards-sk It's all good. I noticed it was from October but couldn't let it slide. :stuck_out_tongue:

I have to wonder though - what do you mean by "prioritized"?

If you mean scheduling it to be worked on, well, that's the fundamental problem. There is too much to do that could be considered a 'top priority'. Here are some of the other ones, almost all of which we've been working on or have been planning for a while: feature branches/pull requests/build matrixes (Spaces), role-based access control (concourse/rfcs#3), Resources v2 (concourse/rfcs#1), reusable tasks (concourse/rfcs#7), improved credential management (concourse/rfcs#5), improved container scheduling, and general resiliency and scalability.

Between all that, it's not surprising that any given GitHub issue with 5 :+1: reactions and no other activity wouldn't be prioritized highly. Even with that being said, there has been work on it when there's been time, in meta-issues like https://github.com/concourse/docker-image-resource/issues/190 to think about the problem at a higher level, and in the two projects I linked to in my previous comment. So really it has been a priority, though I could have definitely done a better job at corralling everything together somewhere more visible.

There is a small team working on Concourse at Pivotal every day, and right now we can work on approximately 3 things at a time (though this may increase in the coming months). We tend to work on the things we determine to be the highest-impact and highest-priority to the most people in order to be as effective as possible, and those tend to take a long time to plan and deliver. Concourse is not an easy project to work on; most of the challenges we tackle don't have a simple solution.

I'm not really sure what else prioritizing would mean, because there isn't a backlog for the community to pull from. Community members are expected to contribute to whatever is a high priority for them, because it's always subjective. We do make an effort to label things with help wanted and good first issue, but there's still no way to dictate what contributors should contribute.

you're not the target of frustration in "Seriously!?"

I get your point, though I don't think that's how other people interpret those comments. It really doesn't matter who the target is; it's not helpful to anyone. It so happens that contributors are more likely to see it.

In any case, I get the frustration - I really do, and I really wish we were in a place where these big issues didn't stick around so long. But I think we need to get there by growing the community and setting an expectation that folks should contribute whenever possible, and make it as easy as possible to get started. This has been my priority for the past year, and we're starting to see results. The upcoming release has more pull requests than we've ever shipped before - and substantial ones, at that.

Contributing doesn't have to be code. Identifying issues that you think should be a high priority and communicating that or keeping the discussion alive is still contributing. I only suggest you do it in a nicer way. :wink:

[eedwards-sk]

You're absolutely right on all counts, and I'm working on improving my tendency to let my own stress bleed over into public forums (which isn't an excuse).

The fix for cache actually is quite useful, since cache_from was in many ways a workaround due to the previous cache behavior.

Thanks for your patience and continued hard work. I'm rooting for concourse even if I'm not the most eloquent at it.

[vito]

@eedwards-sk No worries. Glad to hear the other PR ended up being useful anyway. See you around! 👍