headzoo / surf

Stateful programmatic web browsing in Go.
MIT License
1.48k stars 159 forks source link

Support the ability to use hidden form controls (input, button) #58

Open cloudvant opened 7 years ago

cloudvant commented 7 years ago

Is there a way in Surf to make hidden elements visible so that they can be set and submitted with the form. I'm working with a form that hides/shows fields using javascript and I cannot submit it. Any options:


func (pubmed Pubmed) GetArticles() {
    bow := surf.NewBrowser()
    bow.Open("https://www.ncbi.nlm.nih.gov/pubmed/")

    f, err := bow.Form("form[name='EntrezForm']")
    if err != nil {
        logger.Error(err)
    }
    f.Input("term", QUERY)
    err = f.Submit()
    logger.Error(err)
    logger.Info(bow.Title())

    err = bow.Click("a.tgt_dark:hidden")
    logger.Error(err)
    f, err = bow.Form("form[name='EntrezForm']")
    logger.Error(err)

    f, err = bow.Form("form[name='EntrezForm']")
    err = f.Input("EntrezSystem2.PEntrez.PubMed.Pubmed_ResultsPanel.Pubmed_DisplayBar.SendTo", "File")
    logger.Error(err)
    err = f.Set("EntrezSystem2.PEntrez.PubMed.Pubmed_ResultsPanel.Pubmed_DisplayBar.FFormat", "xml")
    logger.Error(err)
    err = f.Click("EntrezSystem2.PEntrez.PubMed.Pubmed_ResultsPanel.Pubmed_DisplayBar.SendToSubmit")
    logger.Error(err)
    logger.Info(bow.ResponseHeaders())

    file, e := os.Create("./pubmed.xml")
    if e != nil {
        logger.Error(e)
    }
    defer file.Close()
    bow.Download(file)
}
danmux commented 6 years ago

It ignores and css attributes of form elements in the dom tree of the selector.

The field has to have a 'name' attribute - perhaps that is missing, or perhaps the js is actually attaching and removing the field element.

Also it only parses the set given in this list sel.Find("input,button,textarea,select")