search

ovh / venom

🐍 Manage and run your integration tests with efficiency - Venom run executors (script, HTTP Request, web, imap, etc... ) and assertions
Apache License 2.0
1.06k stars 144 forks source link

(bug) Unable to assert test case on venom.datetime #759

Open bcecchinato opened 11 months ago

bcecchinato commented 11 months ago

Hello !

This test fails every time :

---
name: "test"
testcases:
  - name: "Bad RFC"
    steps:
      - name: "Say hello"
        type: exec
        script: echo "Hello !"
        delay: 60
        assertions:
          - result.code MustEqual 0
          - venom.datetime MustHappenOnOrAfter "300 seconds from now"
        retry: 1000
        retry_if:
          - result.code MustEqual 0
          - venom.datetime MustHappenOnOrBefore "300 seconds from now"

The result output is :

 • test (test.yml)
     • Bad-RFC
         • Assert Flip Resistor FAIL
           Testcase "Bad RFC", step #0-0: retry conditions not fulfilled, skipping 1000 remaining retries (test.yml:8)
           Testcase "Bad RFC", step #0-0: Assertion "venom.datetime MustHappenOnOrAfter \"300 seconds from now\"" failed. invalid date provided with %!q(<nil>) not in RFC3339 format or humanize format (test.yml:12)
           Testcase "Bad RFC", step #0-0: At least one required assertion failed, skipping remaining steps (test.yml:8)
           0 other steps were skipped

The variable venom.datetime doesn't look to be populated.

ivan-velasco commented 9 months ago

If you run in verbose do you see a format for venom.datetime such as 2024-02-21T15:30:45Z

@bcecchinato

bcecchinato commented 9 months ago

@ivan-velasco The field looks to be formatted indeed, here is the output of the test :

{
  "test_suites": [
    {
      "name": "test",
      "testcases": [
        {
          "name": "Bad-RFC",
          "vars": {
            "__Len__": "8",
            "__Type__": "Map",
            "venom.datetime": "2024-02-23T08:01:17+01:00",
            "venom.executable": "/usr/local/sbin/venom",
            "venom.libdir": "/etc/venom",
            "venom.outputdir": "/var/log/venom",
            "venom.testcase": "Bad-RFC",
            "venom.testsuite": "test",
            "venom.testsuite.filename": "test.yml",
            "venom.testsuite.filepath": "test.yml",
            "venom.testsuite.name": "test",
            "venom.testsuite.shortName": "test",
            "venom.testsuite.workdir": "/root",
            "venom.timestamp": "1708671677"
          },
          "skip": null,
          "steps": [
            {
              "assertions": [
                "result.code MustEqual 0",
                "venom.datetime MustHappenOnOrAfter \"300 seconds from now\""
              ],
              "delay": 60,
              "name": "Say hello",
              "retry": 1000,
              "retry_if": [
                "result.code MustEqual 0",
                "venom.datetime MustHappenOnOrBefore \"300 seconds from now\""
              ],
              "script": "echo \"Hello !\"",
              "type": "exec"
            }
          ],
          "skipped": null,
          "status": "FAIL",
          "duration": 0.01584637,
          "start": "2024-02-23T08:01:17.799694028+01:00",
          "end": "2024-02-23T08:01:17.815540399+01:00",
          "results": [
            {
              "name": "Say hello",
              "errors": [
                {
                  "value": "Testcase \"Bad RFC\", step #0-0: retry conditions not fulfilled, skipping 1000 remaining retries (test.yml:8)"
                },
                {
                  "value": "Testcase \"Bad RFC\", step #0-0: Assertion \"venom.datetime MustHappenOnOrAfter \\\"300 seconds from now\\\"\" failed. invalid date RFC3339 provided with %!q(\u003cnil\u003e) (test.yml:12)"
                },
                {
                  "value": "Testcase \"Bad RFC\", step #0-0: At least one required assertion failed, skipping remaining steps (test.yml:8)"
                }
              ],
              "skipped": null,
              "status": "FAIL",
              "raw": "YXNzZXJ0aW9uczoKICAgIC0gcmVzdWx0LmNvZGUgTXVzdEVxdWFsIDAKICAgIC0gdmVub20uZGF0ZXRpbWUgTXVzdEhhcHBlbk9uT3JBZnRlciAiMzAwIHNlY29uZHMgZnJvbSBub3ciCmRlbGF5OiA2MApuYW1lOiBTYXkgaGVsbG8KcmV0cnk6IDEwMDAKcmV0cnlfaWY6CiAgICAtIHJlc3VsdC5jb2RlIE11c3RFcXVhbCAwCiAgICAtIHZlbm9tLmRhdGV0aW1lIE11c3RIYXBwZW5Pbk9yQmVmb3JlICIzMDAgc2Vjb25kcyBmcm9tIG5vdyIKc2NyaXB0OiBlY2hvICJIZWxsbyAhIgp0eXBlOiBleGVjCg==",
              "interpolated": "YXNzZXJ0aW9uczoKICAgIC0gcmVzdWx0LmNvZGUgTXVzdEVxdWFsIDAKICAgIC0gdmVub20uZGF0ZXRpbWUgTXVzdEhhcHBlbk9uT3JBZnRlciAiMzAwIHNlY29uZHMgZnJvbSBub3ciCmRlbGF5OiA2MApuYW1lOiBTYXkgaGVsbG8KcmV0cnk6IDEwMDAKcmV0cnlfaWY6CiAgICAtIHJlc3VsdC5jb2RlIE11c3RFcXVhbCAwCiAgICAtIHZlbm9tLmRhdGV0aW1lIE11c3RIYXBwZW5Pbk9yQmVmb3JlICIzMDAgc2Vjb25kcyBmcm9tIG5vdyIKc2NyaXB0OiBlY2hvICJIZWxsbyAhIgp0eXBlOiBleGVjCg==",
              "number": 0,
              "rangedIndex": 0,
              "rangedEnable": false,
              "inputVars": {
                "__Len__": "15",
                "__Type__": "Map",
                "venom.datetime": "2024-02-23T08:01:17+01:00",
                "venom.executable": "/usr/local/sbin/venom",
                "venom.libdir": "/etc/venom",
                "venom.outputdir": "/var/log/venom",
                "venom.testcase": "Bad-RFC",
                "venom.teststep.number": "0",
                "venom.testsuite": "test",
                "venom.testsuite.filename": "test.yml",
                "venom.testsuite.filepath": "test.yml",
                "venom.testsuite.name": "test",
                "venom.testsuite.shortName": "test",
                "venom.testsuite.workdir": "/root",
                "venom.timestamp": "1708671677"
              },
              "computedVars": null,
              "computedInfos": null,
              "assertionsApplied": {
                "ok": false,
                "assertions": [
                  {
                    "assertion": "result.code MustEqual 0",
                    "isOK": true
                  },
                  {
                    "assertion": "venom.datetime MustHappenOnOrAfter \"300 seconds from now\"",
                    "isOK": false
                  }
                ]
              },
              "systemout": "Hello !\n",
              "systemerr": "\n",
              "duration": 0.011183072,
              "start": "2024-02-23T08:01:17.80373463+01:00",
              "end": "2024-02-23T08:01:17.814917963+01:00"
            }
          ]
        }
      ],
      "vars": {
        "__Len__": "8",
        "__Type__": "Map",
        "venom.datetime": "2024-02-23T08:01:17+01:00",
        "venom.executable": "/usr/local/sbin/venom",
        "venom.libdir": "/etc/venom",
        "venom.outputdir": "/var/log/venom",
        "venom.testsuite": "test",
        "venom.testsuite.filename": "test.yml",
        "venom.testsuite.filepath": "test.yml",
        "venom.testsuite.name": "test",
        "venom.testsuite.shortName": "test",
        "venom.testsuite.workdir": "/root",
        "venom.timestamp": "1708671677"
      },
      "shortname": "test",
      "filename": "test.yml",
      "filepath": "test.yml",
      "computed_vars": {
        "Bad-RFC.__type__": "Result",
        "Bad-RFC.result.__len__": 7,
        "Bad-RFC.result.code": "0",
        "Bad-RFC.result.err": "",
        "Bad-RFC.result.systemerr": "",
        "Bad-RFC.result.systemerrjson": "",
        "Bad-RFC.result.systemout": "Hello !",
        "Bad-RFC.result.systemoutjson": "",
        "Bad-RFC.result.timeseconds": 0.008669681
      },
      "workdir": "/root",
      "status": "FAIL",
      "duration": 0.016198239,
      "start": "2024-02-23T08:01:17.799410023+01:00",
      "end": "2024-02-23T08:01:17.815608366+01:00",
      "nbTestcasesFail": 1,
      "nbTestcasesPass": 0,
      "nbTestcasesSkip": 0
    }
  ],
  "status": "FAIL",
  "nbTestsuitesFail": 1,
  "nbTestsuitesPass": 0,
  "nbTestsuitesSkip": 0,
  "duration": 0.016237509,
  "start": "2024-02-23T08:01:17.799376222+01:00",
  "end": "2024-02-23T08:01:17.815613835+01:00"
}
ivan-velasco commented 9 months ago

@bcecchinato If you update the assertion to be 5 minutes from now does that have the same error?

bcecchinato commented 9 months ago

@ivan-velasco same issue :

      [trac] writing /var/log/venom/venom.log
 • test (test.yml)
    • Bad-RFC
        • Say hello FAIL
          Testcase "Bad RFC", step #0-0: retry conditions not fulfilled, skipping 1000 remaining retries (test.yml:8)
          Testcase "Bad RFC", step #0-0: Assertion "venom.datetime MustHappenOnOrAfter \"5 minutes from now\"" failed. invalid date RFC3339 provided with %!q(<nil>) (test.yml:12)
          Testcase "Bad RFC", step #0-0: At least one required assertion failed, skipping remaining steps (test.yml:8)
          0 other steps were skipped
william-hyy commented 7 months ago

I am having the same issue when comparing 2 different dates