search

benibela / xidel

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
http://www.videlibri.de/xidel.html
GNU General Public License v3.0
674 stars 42 forks source link

Cookies header (and other headers) does not work when using "follow". #86

Open Baltazar500 opened 2 years ago

Baltazar500 commented 2 years ago

Cookies header (and other headers) does not work when using "follow". For base url only. Checked via BURP (used --proxy key) I changed the position of the header in the expression before and after follow, but it didn't help.

Cookies from the file did not check, and it is difficult to use them

benibela commented 2 years ago

Indeed, it does not use the header. --method, --data, and --header

Mostly because, when the follow is a POST request, it might need its own headers that would be calculated in the follow option

You could use --follow ' {"header": "....", "url": "..."} '

Or the cookies from the file

Baltazar500 commented 2 years ago

You could use --follow ' {"header": "....", "url": "..."} '

@benibela, Thanks, this works, but each new child or parallel request must be provided with a cookie header. Is it possible to use the specified cookie header at the beginning for all parallel or child requests? Maybe a special option?

benibela commented 2 years ago

You can save the headers in a variable and reuse them later

--follow ' {"header": $myheaders := "....", "url": "..."} '

then

--follow ' {"header": $myheaders, "url": "..."} '

I could change it to share Cookie headers (but not other headers like Referer). Or I could add an option --global-header="Somehader: " that shares the header to the children. Or an option like --make-all-headers-global.

Baltazar500 commented 2 years ago

@benibela,

You can save the headers in a variable and reuse them later

That's what I did then. Only I had to repeat this 6 times and because of this, the expression became much larger

I could change it to share Cookie headers (but not other headers like Referer). Or I could add an option --global-header="Somehader: " that shares the header to the children. Or an option like --make-all-headers-global.

The parameter --global-header="Somehader: " is what is needed in this case :) But the --make-all-headers-global parameter, I think, would also be useful :)