search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47k stars 3.18k forks source link

type / submit in Cypress not triggering HTML validation on elements #6678

Open brettz9 opened 4 years ago

brettz9 commented 4 years ago

Current behavior:

Not finding invalid elements, whether by:

  1. cy.get('<selector>:invalid')
cy.get('[data-name="name"]:invalid').should('have.length', 1);
  1. validity property
cy.get('[data-name="name"]').then(($input) => {
  return expect($input[0].validity.tooShort).to.be.true;
});

Desired behavior:

Correctly indicate that the element is indeed invalid (so that assertions can be based on this).

Test code to reproduce

These tests both fail in showing any problems with invalidity, despite the HTML element being queried having minlength attribute of 3 and our test typing only a single letter ("a"):

it('should catch invalid element by pseudo-selector', () => {
    cy.visit('http://localhost:8000/');
    const tooShortOfAName = 'a';
    cy.get('[data-name="name"]').type(tooShortOfAName);

    // Fails with or without a form click
    // cy.get('[data-name="action2"]').click();

    // Fails here
    cy.get('[data-name="name"]:invalid').should('have.length', 1);
  })

  it('should catch invalid element by JS API', function () {
    cy.visit('http://localhost:8000/');
    const tooShortOfAName = 'a';
    cy.get('[data-name="name"]').type(tooShortOfAName);

    // Fails with or without a form click
    // cy.get('[data-name="action2"]').click();

    // Fails here
    cy.get('[data-name="name"]').then(($input) => {
      return expect($input[0].validity.tooShort).to.be.true;
    });
  });

Here's the minimal repo to reproduce: (note that I've added a cypress install as I think you may want to do in your boilerplate also since your docs refer to installing cypress locally)

https://github.com/brettz9/cypress-test-tiny/tree/not-getting-invalidity

(i.e., on the not-getting-invalidity branch)

Outdated info I've also tacked on an environmental variable test (same as #2605 ?) ```js it('should obtain environmental variables', function () { cy.log(Cypress.env('secret')) expect(Cypress.env('secret')).to.equal('Be good'); }); ``` ...which is not *consistently* getting the `secret` despite it being set in `plugins/index.js` ```js module.exports = (on, config) => { config.env = config.env || {}; config.env.secret = 'Be good'; return config; ); ``` ...consistent with the approach described for dynamically setting the environmental variables at https://docs.cypress.io/api/plugins/configuration-api.html#Switch-between-multiple-configuration-files or https://docs.cypress.io/api/cypress-api/env.html#From-a-plugin.

Versions

4.1.0, Mac OSX 10.15.2, Chrome 80.0.3987.132 (Official Build) (64-bit)

brettz9 commented 4 years ago

I might add my validity checks are akin to that added in https://github.com/cypress-io/cypress-documentation/pull/1919 (at https://docs.cypress.io/faq/questions/using-cypress-faq.html#Can-I-check-that-a-form%E2%80%99s-HTML-form-validation-is-shown-when-an-input-is-invalid )

brettz9 commented 4 years ago

My apologies re: the environmental issue; it seems that my issue there may have been not returning the config object. However, the main problem with validity checking remains.

jennifer-shehane commented 4 years ago

So the problem here is not that it's not finding elements with a psuedo-selector of invalid, it's that the actions being performed through Cypress are not triggering the validation checks on the form for some reason.

You can use jQuery to query for input:invalid on the application under test after Cypress performs the type and even the click to submit the form and see that it finds no invalid elements.

Reproducible Example

spec.js

it('does not find invalid input', () => {
  cy.visit('index.html')
  cy.get('input').type('a')
  // cy.contains('Update').click()
  cy.get('input:invalid').should('have.length', 1)
})

index.html

<!DOCTYPE html>
<body>
    <input minlength="3">
    <button type="submit">Update</button>
</body>
</html>
Screen Shot 2020-03-10 at 2 01 45 PM

Events triggered during type.

Screen Shot 2020-03-10 at 1 51 42 PM

If you type manually into the form while the test is running, it will all of a sudden pass.

Screen Shot 2020-03-10 at 2 04 08 PM
shamgang commented 4 years ago

In case it helps: I have 2 forms, one with a text and password input, and one with a text, password, and numerical input. In both cases only the text input exhibits this bug. The password and numerical inputs behave as expected - form validation popups are appearing. This remains true if I reorder the inputs on the page and change the order I type into them in Cypress. Node 12.13.0 Cypress 4.8.0 Chrome 83.0.4103.97 Windows 10 Home v1903 build 18362.900

zedrdave commented 4 years ago

Just to add my own experience: getting this bug on a minlength validation on a password input: when using cy.get().type() to type text that should be too short, I am able to submit the form both manually and through Cypress.

In essence, it looks as if Cypress is not sending the necessary key events for the form to perform validation.

shin10 commented 4 years ago

I came across this issue as well. The minlength attribute will be ignored. Validation of the pattern attribute pattern=".{6}" however will be triggered. Unfortunately the resulting message ("Please match the requested format") is very vague and rather confusing for the user. Further adding setCustomValidity manually wouldn't be my favourite option.

sainthkh commented 4 years ago

I tried form.reportValidity(). But somehow, it doesn't work. It always return true when there's an error in the form.

yanfengliu commented 3 years ago

According to MDN, minLength is only triggered if value is changed by user.

brettz9 commented 3 years ago

Is this planned, e.g., on a Roadmap, or given a priority? It is a pretty fundamental need to be able to check forms, testing for validity (for those using the standard mechanisms for doing so). Thanks in advance...

sainthkh commented 3 years ago

@brett9 Unfortunately, it's hard to find browser API that triggers form validation. Maybe, this can be solved with native events.

brettz9 commented 3 years ago

@sainthkh : Thanks for the info and idea. If this would be best addressed by Native Browser Events, I would hope its consideration might be added to the list on #311 , @brian-mann .

bahmutov commented 3 years ago

An update: for me the validations are all working, you can see in https://glebbahmutov.com/cypress-examples/6.4.0/recipes/form-validation.html

Note: cy.submit does not check the validity of the form before submitting https://github.com/cypress-io/cypress/issues/14911 while clicking on the submit button does check

cypress-bot[bot] commented 3 years ago

The code for this is done in cypress-io/cypress#14965, but has yet to be released. We'll update this issue and reference the changelog when it's released.

This change was reverted in https://github.com/cypress-io/cypress/pull/15302

brettz9 commented 3 years ago

Sorry for the delay in replying...

As one example, I have:

<input type="password" autocomplete="new-password" class="form-control" required="" name="pass" id="pass-tf" data-name="reset-pass" minlength="6">

but this code fails:

      const tooShortPassword = 'a';
      cy.get('[data-name="reset-pass"]').type(tooShortPassword);

      cy.get('[data-name="reset-pass"]:invalid').should('have.length', 1); // Should be 1, but gives 0

The minimal test repo in the original post covers this and the other cases I believe are still occurring.

I.e., it's not just about the submission.

brettz9 commented 3 years ago

14965 seems not to actually be covering this issue. My concern is before submit, so I think this issue should still be open.

jennifer-shehane commented 3 years ago

@brettz9 Yeah, you're right. I'm actually not able to get the original example to work with the new implementation. https://github.com/brettz9/cypress-test-tiny/tree/not-getting-invalidity?rgh-link-date=2020-03-09T04%3A34%3A03Z cc @sainthkh

  1. I don't see .submit() failing the test as the PR described it should.
  2. I don't see a way to check the conditions of the validity after submitting the form. How do you verify that the correct message is displayed for an invalid field in the example?
sainthkh commented 3 years ago

Confirmed that #14965 didn't fix the issue.

But the problem is that it cannot be fixed now. Because all validations are not created equal in browsers. As for the validations like required, mix, min, they always work whether it's a user input or not. But for those like minlength, maxlength, they are only triggered when users input the value.

If you want to test it yourself, sample code is below:

<input id="x" minlength="3" value="a" />
<input id="y" required />
cy.get('#x').then($x => expect($x[0].checkValidity()).to.be.false) // => fail
cy.get('#y').then($y => expect($y[0].checkValidity()).to.be.false) // => succeed

Native events might solve the problem. But I'm not sure about that.

tusharAdvanced commented 2 years ago

Is there any solution on this ? I am facing the same.

a-bashkatov commented 1 year ago

I could solve this problem using "realType" from cypress-real-events, then the input field is validated as with real text input. However, this solution is not suitable, for example, for Firefox.