[Workload] State test: Does the CNF use a non-cloud native data store: memory volumes found?

[lixuna]

Acceptance Criteria - TBD, to be based on 159

Statelessness test: Does the CNF use a non-cloud native data store: memory volumes found?

Short Description:

• Check for non-cloud native data stores
• Does the CNF use memory volumes?

Reference:

• https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
• https://docs.docker.com/storage/tmpfs/ (used with emptydir)
• https://www.alibabacloud.com/blog/kubernetes-volume-basics-emptydir-and-persistentvolume_594834

Test Category:

• Stateless (eg. Cloud native State)

Type of test (static or runtime)

• Runtime

Scoring:

• tbd

Are there any pre-requisites?

• tbd

Command name:

• tbd

---

Tasks:

Proof of concept tasks:

• [ ] Add a quick overview of the type of test you are researching/creating a POC for
• [ ] Investigate potential tools for implementation and document findings, https://hackmd.io/new (add link to ticket)
• [ ] Select a tool to use, minimal/least effort, and add selection to ticket
• [ ] Add new POC test code
• [ ] Tag 1 or more people to peer review

Code implementation tasks:

• [ ]
• [ ]

Documentation tasks:

• [ ] Update Test Categories md if needed
• [ ] Update USAGE md if needed
• [ ] Update installation instructions if needed

---

QA tasks

Dev Review:

• [ ] walk through A/C
• [ ] do you get the expected result?
• [ ] if yes,
  • [ ] move to Needs Peer Review column
  • [ ] create Pull Request and follow check list
  • [ ] Assign 1 or more people for peer review
• [ ] if no, document what additional tasks will be needed

Peer review:

• [ ] walk through A/C
• [ ] do you get the expected result?
• [ ] if yes,
  • [ ] move to Reviewer Approved column
  • [ ] Approve pull request
• [ ] if no,
  • [ ] document what did not go as expected, including error messages and screenshots (if possible)
  • [ ] Add comment to pull request
  • [ ] request changes to pull request

[taylor]

DRAFT Acceptance Criteria: building on A/C from #159 and #269

Acceptance Criteria for peer review:

--

Documentation is updated:

• [ ] Documentation lists new test name
• [ ] Test Categories md is updated
• [ ] USAGE md is updated
• [ ] Installation instructions is updated

Common steps

• [ ] Find or create a K8s cluster to use
  • Talk with team for access to an existing cluster
  • Kind K8s clusters will work as well
• [ ] Set the KUBECONFIG environment variable for accessing the target K8s cluster
• [ ] As a developer, Pull down latest version of CNF conformance test code from {FEATURE} branch
  • [ ] Check that {FEATURE} Branch passes all A/C
  • [ ] Check that {FEATURE} Branch passes tests in Travis CI
• [ ] As a peer reviewer, check that PR passes Travis CI tests
  • [ ] if yes, merge PR to master
  • [ ] if no, make a comment and request that developer make changes to pass tests
• [ ] As a peer reviewer, Pull down latest version of CNF conformance test code from master branch
• [ ] Run shards install

Spec test should pass

• [ ] Run crystal spec
• [ ] I expect to see PASSED for the tests

Update Prerequistite checker

• N/A

Update example CNFs cnf-conformance.yml, if needed

• [ ] continue, If there is an update to cnf-conformance.yml
• [ ] for each CNF Example
• [ ] Update the example CNF's cnf-conformance.yml to include the new changes
• [ ] Update the example CNF's README if needed
• [ ] Test the example CNF following it's README

Update yml validator, if needed

• N/A

When using the test cnf-conformance volume_memory_emptydir_found command the sample "good" CNF should PASS test successfully:

• [ ] Run ./cnf-conformance setup
• [ ] Run ./cnf-conformance sample_coredns_setup
• [ ] Run ./cnf-conformance volume_memory_emptydir_found
• [ ] I expect to see the output and results from the test
• [ ] I expect to see PASSED for the test. Example:
  • [ ] "✔️ PASSED: memory backed emptyDir found [EMOJI]"

When using the test crystal spec --tagvolume_memory_emptydir_found`` command the sample "good" CNF should PASS and the "bad example CNF" should FAIL test:

• [ ] Run crystal spec --tag volume_localstorage_not_found
• [ ] I expect to see the output and results from the test
• [ ] I expect to see "✔️ PASSED: memory backed emptyDir found [EMOJI]
• [ ] I expect to see "❌ FAIL: ???? [EMOJI]"

I would expect the CNF to pass when I run the entire test category suite (statelessness)

• [ ] Follow common steps above
• [ ] Run ./cnf-conformance sample_coredns_setup
• [ ] Run ./cnf-conformance statelessness
• [ ] I expect to see the output and results from all tests in the test category
• [ ] I expect to see PASSED for the test and test category

I would expect the CNF to pass when I run the entire test suite (all)

• [ ] Follow common steps above
• [ ] Run ./cnf-conformance sample_coredns_setup
• [ ] Run ./cnf-conformance all
• [ ] I expect to see the output from the test
• [ ] I expect to see PASSED for the test

Screenshots of new test and results in the comments

• [ ] New test should be added to the points file
• [ ] New test score should be added to the final score
• [ ] I can see a screenshot of the new test in the comments of this ticket

[taylor]

@denverwilliams @wavell this looks like a test that would give positive points if found and 0 points if not found. memory backed emptyDir means the application should be designed to handle that temporary storage disappearing if the pod is "deleted" or the node goes down.

Not using memory backed storage does not indicate anything is wrong.

I do not see the counter where using memory backed storage would be negative regarding cloud native principles.

Reference:

• https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
• https://docs.docker.com/storage/tmpfs/ (used with emptydir)
• https://www.alibabacloud.com/blog/kubernetes-volume-basics-emptydir-and-persistentvolume_594834

[wvwatson]

https://hackmd.io/qd10889UQIisxCO_E3VsuQ

[wvwatson]

1) We need to check if you are expecting the memory volumes to be persistent a) if memory volumes are expected to be persistent, and the user is using non-cloud native memory volumes then fail test 2) (alternative) If ignoring (1) and testing the integrity of the system by reseting a memory volume node, we will need to have a way to see if the data is corrupted. a) this would require something more than the readiness and liveness tests (requires and integration or probe test)

[wvwatson]

Points awarded if the memory volume is mounted and no other volume is mounted. This ensures that the volume is being used

[wvwatson]

Test outcome: If your application is using in memory volumes, is it stateless?

[wvwatson]

942 related

[denverwilliams]

Proposal to reject memory volume test.

After a developer planning call to research the feasibility of test we came to the conclusion it would be low value based on the following points.

• 1. A test that checks if a cnf has a memory volume alone wouldn't provide much value as it doesn't really prove that the cnf is doing anything we would consider a 'best-practice'
     • There are no inherent benefits that are directly observable to using, or not using, a memory volume. A completely stateless cnf would be considered a good thing, but the presence of a memory volume doesn't prove this, it's only suggestive of it. The memory volume might simply get created and not used.
     • The inverse is also true. A cnf that doesn't have a memory volume doesn't prove that it's not stateless. For example, without a memory volume you can still write any necessary files directly to disk inside the cnf's filesystem. These files would still be ephemeral and will be removed when the cnf is deleted or lost when scheduled to another node.
     • You can test this with: docker run --name test -ti debian:latest /bin/bash echo 'test' > test.file exit docker start test docker exec -ti test cat /test.file You'll notice that the data is still there, so by default 'without' a memory volume you can still write temporary files that will survive a Pod reboot, but be lost on the Pod deletion or when rescheduled.
• 2. We discussed the idea adding an integrity check to the test to verify that the memory volume is being used for caching. A test that checks for this would be a valuable test, but unfortunately, there isn't a feasible way to implement it.
     • A cnf that has a memory volume alone isn't enough to prove it does caching, and would be too easily gameable for 'free points' if we grant points based on the existance of the volume alone. We would have to introspect the data on the memory volume somehow to see if it's being used as a legitimate cache store, and there doesn't seem to be a feasible way of doing this in a timely manner.
     • The other catch is the lack of a memory volume doesn't prove that caching is not done. Ephemeral cache data could be written and stored in the cnfs root filesystem instead. As shown in the above code example, it would be ephemeral and removed on Pod deletion or rescheduling.
• 3. This test seems low value when compared to other tests we have implemented or looked at implementing. The use of caching seems like less of a 'Cloud-Native' best practice and more like a standard development practice. The test suite will probably provide more value if we focus on tests that check for less understood or more contentious best-practices that are more specific to the 'cloud-native' ethos.

[taylor]

Agreed to close this. cc: @lixuna