Can't load resource from file - multiple resources found

[ghost]

Hi @edaniszewski, me again...

Now that I can load my manifest I get the following message:

ValueError: Unable to load resource from file - multiple resources found in specified file.

This will prove to be an issue as the majority of our manifests contain numerous resources, for example see my demo manifest below:

apiVersion: v1
kind: Service
metadata:
  name: redacted
spec:
  ports:
  - name: web
    port: 80
    protocol: TCP
    targetPort: redacted
  selector:
    deployment: kubetest-base-app
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: redacted
spec:
  ports:
  - name: web
    port: 80
    protocol: TCP
    targetPort: redacted
  selector:
    deployment: kubetest-base-app
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: redacted
spec:
  ports:
  - name: web
    port: 80
    protocol: TCP
    targetPort: redacted
  selector:
    deployment: kubetest-base-app
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubetest-base-app
  labels:
    deployment: kubetest-base-app
spec:
  replicas: 1
  selector:
    matchLabels:
      deployment: kubetest-base-app
  template:
    metadata:
      labels:
        deployment: kubetest-base-app
    spec:
      containers:
        - name: app
          image: redacted
          imagePullPolicy: Always
          ports:
          - name: redacted
            containerPort: 5000

It's only the deployment that I need to use from a kubetest perspective, the other resource is for tavern tests.

Have I missed something here or this another feature request?

Thanks

[edaniszewski]

Thanks for opening the issue! I don't quite remember if I implemented handling for multiple resources in a file, or if the farthest I got for it was just planning out the feature... I'll dig through the code in a little bit later today and report back.

If it is implemented, I don't remember considering the use case of applying/creating a subset of resources from a given manifest, but I think support for that could be added fairly easily.

[ghost]

I vaguely remember reading in the docs that kubetest looks for the the first deployment in the manifest.

[ghost]

Hey @edaniszewski, I hope you're well!

I don't suppose you've had a chance to look at this at all please?

Many thanks, Gavin

[edaniszewski]

Sorry for the delay.. got bogged down a bit last week and this slipped under my radar. I looked at this over the weekend and it looks like it was not allowing multiple definitions in a loaded manifest file, which is not ideal. I made some changes and opened up a PR (#175) which (I think) should fix this issue.

If you wouldn't mind looking it over and letting me know if the changes seem like they would address this issue, it would be much appreciated! (:

[ghost]

No worries, thank you so much for taking the time to take a look.

Sure, I'll have a look now.

Again, thanks for your time and effort in helping me out!

[edaniszewski]

No problem, happy to do it!

The PR has been merged in and is part of the 0.6.4 release. Thanks for opening the issue for it. I'll close this out now, but if anything comes up relating to this, it can be reopened, or a new issue opened up.

[ghost]

@edaniszewski, Not sure if I've missed something or not, but I can't get the changes to work.

After reviewing your demo test it appears that no changes need to be made to my test (correct me if I'm wrong). But upon running the test the container times out (even after increasing to 180 secs).

Could you advise please?

Manifest is still as above, test currently looks like this:

import pytest

def test_app_readiness(kube):
    deployment = kube.load_deployment('/kubetest-base/utils/kubernetes-manifests/kubetest-base-app/kubetest-base-app.yaml')
    deployment.create()

    deployment.wait_until_ready(timeout=180)

    pods = deployment.get_pods()
    assert len(pods) == 1

    pod = pods[0]
    pod.wait_until_ready(timeout=180)
    containers = pod.get_containers()
    assert len(containers) == 1

This is the output I get:

============================= test session starts ==============================
platform linux -- Python 3.7.7, pytest-4.5.0, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
kubetest config file: in-cluster
kubetest context: current context
rootdir: /kubetest-base, inifile: pytest.ini, testpaths: tests
plugins: tavern-0.34.0, kubetest-0.6.4
collecting ... collected 4 items

Unable to cache logs for kubetest-base-app-6d6bd945b9-6cppv::app ((400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '232', 'Content-Type': 'application/json', 'Date': 'Wed, 25 Mar 2020 07:06:44 GMT'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"app\" in pod \"kubetest-base-app-6d6bd945b9-6cppv\" is waiting to start: trying and failing to pull image","reason":"BadRequest","code":400}

)
tests/test_app-readiness.py::test_app_readiness FAILED                   [ 25%]
tests/test_app-status.tavern.yaml::ensure the tavern base test server throws a 200 status using the admin host PASSED [ 50%]
tests/test_app-status.tavern.yaml::ensure the tavern base test server throws a 200 status using the mtls host PASSED [ 75%]
tests/test_app-status.tavern.yaml::ensure the tavern base test server throws a 200 status using the tenant host PASSED [100%]

=================================== FAILURES ===================================
______________________________ test_app_readiness ______________________________

kube = <kubetest.client.TestClient object at 0x7f04345f7990>

    def test_app_readiness(kube):
        deployment = kube.load_deployment('/kubetest-base/utils/kubernetes-manifests/kubetest-base-app/kubetest-base-app.yaml')
        deployment.create()

>       deployment.wait_until_ready(timeout=180)

tests/test_app-readiness.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.7/site-packages/kubetest/objects/api_object.py:157: in wait_until_ready
    fail_on_api_error=fail_on_api_error,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

condition = <Condition (name: api object ready, met: False)>, timeout = 180
interval = 1, fail_on_api_error = False

    def wait_for_condition(
            condition: Condition,
            timeout: int = None,
            interval: Union[int, float] = 1,
            fail_on_api_error: bool = True,
    ) -> None:
        """Wait for a condition to be met.

        Args:
            condition: The Condition to wait for.
            timeout: The maximum time to wait, in seconds, for the condition to be met.
                If unspecified, this function will wait indefinitely. If specified and
                the timeout is met or exceeded, a TimeoutError will be raised.
            interval: The time, in seconds, to wait before re-checking the condition.
            fail_on_api_error: Fail the condition checks if a Kubernetes API error is
                incurred. An API error can be raised for a number of reasons, including
                a Pod being restarted and temporarily unavailable. Disabling this will
                cause those errors to be ignored, allowing the check to continue until
                timeout or resolution. (default: True).

        Raises:
            TimeoutError: The specified timeout was exceeded.
        """
        log.info(f'waiting for condition: {condition}')

        # define the maximum time to wait. once this is met, we should
        # stop waiting.
        max_time = None
        if timeout is not None:
            max_time = time.time() + timeout

        # start the wait block
        start = time.time()
        while True:
            if max_time and time.time() >= max_time:
                raise TimeoutError(
>                   f'timed out ({timeout}s) while waiting for condition {condition}'
                )
E               TimeoutError: timed out (180s) while waiting for condition <Condition (name: api object ready, met: False)>

/usr/local/lib/python3.7/site-packages/kubetest/utils.py:122: TimeoutError
------------------------------ Captured log setup ------------------------------
INFO     kubetest:manager.py:353 creating test meta for tests/test_app-readiness.py::test_app_readiness
INFO     kubetest:namespace.py:60 creating namespace "kubetest-test-app-readiness-1585119824"
DEBUG    kubetest:namespace.py:61 namespace: {'api_version': None,
 'kind': None,
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': None,
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': None,
              'managed_fields': None,
              'name': 'kubetest-test-app-readiness-1585119824',
              'namespace': None,
              'owner_references': None,
              'resource_version': None,
              'self_link': None,
              'uid': None},
 'spec': None,
 'status': None}
WARNING  kubetest:api_object.py:112 unknown version (None), falling back to preferred version
------------------------------ Captured log call -------------------------------
INFO     kubetest:client.py:187 loading deployment from path: /kubetest-base/utils/kubernetes-manifests/kubetest-base-app/kubetest-base-app.yaml
INFO     kubetest:deployment.py:111 creating deployment "kubetest-base-app" in namespace "kubetest-test-app-readiness-1585119824"
DEBUG    kubetest:deployment.py:112 deployment: {'api_version': 'apps/v1',
 'kind': 'Deployment',
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': None,
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': {'deployment': 'kubetest-base-app',
                         'kubetest/deployment': '4c45f1e0-f391-42f5-8090-47bbc72ef50c'},
              'managed_fields': None,
              'name': 'kubetest-base-app',
              'namespace': 'kubetest-test-app-readiness-1585119824',
              'owner_references': None,
              'resource_version': None,
              'self_link': None,
              'uid': None},
 'spec': {'min_ready_seconds': None,
          'paused': None,
          'progress_deadline_seconds': None,
          'replicas': 1,
          'revision_history_limit': None,
          'selector': {'match_expressions': None,
                       'match_labels': {'deployment': 'kubetest-base-app',
                                        'kubetest/deployment': '4c45f1e0-f391-42f5-8090-47bbc72ef50c'}},
          'strategy': None,
          'template': {'metadata': {'annotations': None,
                                    'cluster_name': None,
                                    'creation_timestamp': None,
                                    'deletion_grace_period_seconds': None,
                                    'deletion_timestamp': None,
                                    'finalizers': None,
                                    'generate_name': None,
                                    'generation': None,
                                    'initializers': None,
                                    'labels': {'deployment': 'kubetest-base-app',
                                               'kubetest/deployment': '4c45f1e0-f391-42f5-8090-47bbc72ef50c'},
                                    'managed_fields': None,
                                    'name': None,
                                    'namespace': None,
                                    'owner_references': None,
                                    'resource_version': None,
                                    'self_link': None,
                                    'uid': None},
                       'spec': {'active_deadline_seconds': None,
                                'affinity': None,
                                'automount_service_account_token': None,
                                'containers': [{'args': None,
                                                'command': None,
                                                'env': None,
                                                'env_from': None,
                                                'image': 'dev-harbor.cam.zeus.com/gjohnson/kubetest-base-app',
                                                'image_pull_policy': 'Always',
                                                'lifecycle': None,
                                                'liveness_probe': None,
                                                'name': 'app',
                                                'ports': [{'container_port': 5000,
                                                           'host_ip': None,
                                                           'host_port': None,
                                                           'name': 'ive-web',
                                                           'protocol': None}],
                                                'readiness_probe': None,
                                                'resources': None,
                                                'security_context': None,
                                                'stdin': None,
                                                'stdin_once': None,
                                                'termination_message_path': None,
                                                'termination_message_policy': None,
                                                'tty': None,
                                                'volume_devices': None,
                                                'volume_mounts': None,
                                                'working_dir': None}],
                                'dns_config': None,
                                'dns_policy': None,
                                'enable_service_links': None,
                                'host_aliases': None,
                                'host_ipc': None,
                                'host_network': None,
                                'host_pid': None,
                                'hostname': None,
                                'image_pull_secrets': None,
                                'init_containers': None,
                                'node_name': None,
                                'node_selector': None,
                                'preemption_policy': None,
                                'priority': None,
                                'priority_class_name': None,
                                'readiness_gates': None,
                                'restart_policy': None,
                                'runtime_class_name': None,
                                'scheduler_name': None,
                                'security_context': None,
                                'service_account': None,
                                'service_account_name': None,
                                'share_process_namespace': None,
                                'subdomain': None,
                                'termination_grace_period_seconds': None,
                                'tolerations': None,
                                'volumes': None}}},
 'status': None}
INFO     kubetest:utils.py:109 waiting for condition: <Condition (name: api object ready, met: False)>
---------------------------- Captured log teardown -----------------------------
INFO     kubetest:namespace.py:79 deleting namespace "kubetest-test-app-readiness-1585119824"
DEBUG    kubetest:namespace.py:80 delete options: {'api_version': None,
 'dry_run': None,
 'grace_period_seconds': None,
 'kind': None,
 'orphan_dependents': None,
 'preconditions': None,
 'propagation_policy': None}
DEBUG    kubetest:namespace.py:81 namespace: {'api_version': 'v1',
 'kind': 'Namespace',
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2020, 3, 25, 7, 3, 44, tzinfo=tzlocal()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': None,
              'managed_fields': None,
              'name': 'kubetest-test-app-readiness-1585119824',
              'namespace': None,
              'owner_references': None,
              'resource_version': '7300',
              'self_link': '/api/v1/namespaces/kubetest-test-app-readiness-1585119824',
              'uid': 'c3e6604d-6e66-11ea-965d-005056a62833'},
 'spec': {'finalizers': ['kubernetes']},
 'status': {'phase': 'Active'}}
--------- generated xml file: /kubetest-base/reports/test-results.xml ----------
===================== 1 failed, 3 passed in 197.87 seconds =====================
~/Projects/kubetest-demo
SUCCESS: Test results are in kubetest-base/reports

Many thanks!

[edaniszewski]

You're right that no changes need to be made to the test to use the latest updates to manifest load -- it looks like those changes worked since it now appears that it is finding and loading the deployment from that manifest file based on the logs.

Towards the top of the test output I'm seeing this:

Unable to cache logs for kubetest-base-app-6d6bd945b9-6cppv::app ((400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Length': '232', 'Content-Type': 'application/json', 'Date': 'Wed, 25 Mar 2020 07:06:44 GMT'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"app\" in pod \"kubetest-base-app-6d6bd945b9-6cppv\" is waiting to start: trying and failing to pull image","reason":"BadRequest","code":400}

so to me, this sounds like the reason wait_until_ready is failing is because the deployment never gets to that ready state since it can't pull its image. That's my best guess right now. I'll still poke around in the code to see if I can find anything funky in there. It'd also be good to somehow improve the error message output from kubetest since right now its kinda just a big dump of data and can be difficult to parse through.

Let me know if it looks like the image pull was the issue or not - if not, I can dig deeper and see what else I can find.

[ghost]

Hey, thanks for the response.

I get the same message as you above. But kubetest-base-app runs as expected as soon as the pod enters a state of running.

Please see below for logs (the 3 requests made are from the successful Tavern tests):

╰─ kl kubetest-base-app-784474994f-ktrb8
 * Serving Flask app "kubetest-base-app.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 142-072-699
10.42.0.6 - - [25/Mar/2020 19:46:45] "GET / HTTP/1.1" 200 -
10.42.0.6 - - [25/Mar/2020 19:46:50] "GET / HTTP/1.1" 200 -
10.42.0.6 - - [25/Mar/2020 19:46:55] "GET / HTTP/1.1" 200 -

[edaniszewski]

Interesting.. so the deployment does run? It makes sense that it would run as expected once the pod enters the running state as that should unblock the waiter function, but since it doesn't seem like the waiter is unblocking and given the error message that it was unable to pull the image for the deployment, I'm not sure how the kubetest-base-app pod becomes accessible?

[ghost]

I've watched the namespaces in the cluster and found that the Kubetest namespace has issues (there are redactions):

NAMESPACE                                NAME                                      READY   STATUS         RESTARTS   AGE
default                                  kubetest-base-app-784474994f-q5g4b        1/1     Running        0          14s
default                                  kubetest-base-tests-fbg9z                 1/1     Running        0          13s
kube-system                              coredns-695688789-bns7g                   1/1     Running        0          136m
kubetest-test-app-readiness-1585219942   kubetest-base-app-7ffcb8d759-nrg2t        0/1     ErrImagePull   0          8s

On the other hand kubetest-base-app appears to have no issues, see logs from kubectl describe pod (again redactions here):

Name:           kubetest-base-app-784474994f-q5g4b
Namespace:      default
Priority:       0
Node:           gjohnson-03.redacted/10.62.168.179
Start Time:     Thu, 26 Mar 2020 10:52:17 +0000
Labels:         deployment=kubetest-base-app
                pod-template-hash=784474994f
Annotations:    <none>
Status:         Running
IP:             10.42.3.8
IPs:            <none>
Controlled By:  ReplicaSet/kubetest-base-app-784474994f
Containers:
  app:
    Container ID:   containerd://de98ff3f864666b8ad1045e62711f69b67ec9d01b43a4e1d8fa65966aaa74f65
    Image:          redacted/gjohnson/kubetest-base-app:latest-gjohnson
    Image ID:       redacted/gjohnson/kubetest-base-app@sha256:96253cb78249fd513daaa4d8013027e7772596421616e83555f45c098378d557
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 26 Mar 2020 10:52:18 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mlzgz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-mlzgz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mlzgz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                               Message
  ----    ------     ----   ----                               -------
  Normal  Scheduled  5m20s  default-scheduler                  Successfully assigned default/kubetest-base-app-784474994f-q5g4b to gjohnson-03.redacted
  Normal  Pulling    5m18s  kubelet, gjohnson-03.redacted  Pulling image "redacted/gjohnson/kubetest-base-app:latest-gjohnson"
  Normal  Pulled     5m18s  kubelet, gjohnson-03.redacted  Successfully pulled image "redacted/gjohnson/kubetest-base-app:latest-gjohnson"
  Normal  Created    5m18s  kubelet, gjohnson-03.redacted  Created container app
  Normal  Started    5m18s  kubelet, gjohnson-03.redacted  Started container app

Please let me know if you need anything else.

[edaniszewski]

Okay, so to me it seems like when it runs in the default namespace, things appear to work, but when run in any of the auto-generated kubetest namespaces, the image pull error comes up.

It also looks like the app running in the default namespace has a token mounted from a service account, whereas the log output from a few comments up detailing the deployment spec for the instance run in the test namespace doesn't have any service account or security context defined. I'm wondering if you are using an image hosted in a private registry which needs some kind of auth/token to pull? If so, I think that explains why it would be working in the default namespace, but not in the test namespace.

Even if thats not the case, I could see that being a possible use case, and I don't believe kubetest currently supports adding service accounts/secrets to all tests. One could be added by manually defining the manifests for it, but it would have to be included for every test, so it seems like a good user experience improvement for there to be a way to have kubetest apply a service account/something like that to all tests.

[ghost]

My image repo is public, but I have a service account in the kubetest-base-tests manifest (as recommended in my previous issue here see below for service account (incase that makes a difference):

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: null
  name: test-kube-user
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: test-cluster-admin-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: test-kube-user
    namespace: default
---
apiVersion: batch/v1
kind: Job
metadata:
  name: kubetest-base-tests
  namespace: default
  labels:
    app: kubetest-base-tests
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        product: kubetest-base
        app: kubetest-base-tests
        job-owned: "true"
    spec:
      serviceAccountName: test-kube-user
      restartPolicy: Never
      containers:
      - name: kubetest-base-tests
        image: redacted/gjohnson/kubetest-base-tests
        imagePullPolicy: Always
        env:
        - name: PRINT_JUNIT_XML
          value: "1"

Again, 10923045923 thanks for your help with this.

[edaniszewski]

Yeah, if the image is public, then I don't think a service account or pull secret would be the thing causing it. Out of curiosity, what is the output of a kubectl describe on the kubetest-base-app pod running in the kubetest namespace? I'm wondering if there is any additional information in there about why its failing to pull the image.

No prob for the help, I'm glad to (:

[ghost]

Is this here?

[edaniszewski]

It looks like the describe output there is for the kubetest-base-app running in the default namespace, not the one running in the kubetest-generated namespace

[ghost]

This is all I could get I'm afraid:

kubectl describe namespaces kubetest-test-app-readiness-1585327652
Name:         kubetest-test-app-readiness-1585327652
Labels:       <none>
Annotations:  <none>
Status:       Active

No resource quota.

No resource limits.

[edaniszewski]

So it looks like the above is the output for describing the namespace, not the pod, which is where the event info would be for stuff related to the deployment (e.g. like if its failing to pull an image or something).

In saying that, I'm realizing how its probably difficult to actually get the info out because the namespace is auto-generated and once a test fails, its automatically cleaned up, so there may not be a lot of time to go in and inspect whats going on with the deployment. I'm thinking that there may be a better way to capture the describe output for failed tests/pods in the pytest output so one doesn't have to go manually poking around for it.

I think until that gets in, the next best thing is to just add a long sleep in the test case to keep the resources up, which should give enough time to look at the describe detail for the pod.

I've scanned through the bits of the kubetest code that feel pertinent, but I don't see anything there that looks to be the cause of this, so I suspect that its a configuration issue or there is something about the deployment environment which kubetest isn't accounting for that I haven't been able to put my finger on just yet

[ghost]

Lovely, thanks I'll have a look through my code and let you know what comes out of it.