Additional downloaders for `wick-get-url`

[fidian]

There's curl and wget. What about other tools or command-line browsers such as links or lynx? Can nc do it? What about bash doing it itself without any extra binaries?

[fidian]

Need to write a URL parser to get hostname and port and to detect protocol.

If http, both netcat and socat can be used to get the raw body.

wick-get-raw-request() {
    echo -e "GET $PATH HTTP/1.0\r\nHost: $HOST\r\n\r\n"
}

wick-parse-raw-response() {
    # if sed is installed
    echo "$1" | sed -E '0,/^\r?$/ d'

    # If no sed
    BREAK=$'\n\r\n'
    echo "${1#*$BREAK}"
}

RAW=$(wick-get-raw-request | socat TCP:$HOST:$PORT STDIO)
RAW=$(wick-get-raw-request | nc -q $TIMEOUT -w $TIMEOUT $HOST $PORT)

wick-parse-raw-response RAW

Not sure how well these work with binary data or multiple megabytes of information. Probably not too well.

[fidian]

sed without -E

sed '0,/^\r\{0,1\}$/d' data.txt