Hexxeh / rpi-update

An easier way to update the firmware of your Raspberry Pi
MIT License
1.94k stars 232 forks source link

Use jq instead of awk for the reading of json data from Github API #268

Closed dbyio closed 6 years ago

dbyio commented 6 years ago

Github seems to have changed the json output format of its API endpoints (removing newlines) which breaks the current curl pipe awk parsing used currently to handle a number of things. It's likely this could be fixed with awk, but I think a json-native parsing makes things more durable. Please note that this change makes the jq utility a requirement for using rpi-update.

popcornmix commented 6 years ago

Thanks for the PR. Requiring a new package to be installed before rpi-update works again is a definite downside. This will need some thought.

popcornmix commented 6 years ago

I've just had a look and all four places you've changed seem to be giving the correct responses for me here (with existing awk parsing). Is the changed newline format still visible for you?

dbyio commented 6 years ago

It is:

$ curl -Ls https://api.github.com/repos/Hexxeh/rpi-firmware/commits/master | awk '{print $1}'
{"sha":"27ed1a386e52c24c9eb76b7dd7306a7016fb6669","node_id":"MDY6Q29tbWl0NDA5NTgxMToyN2VkMWEzODZlNTJjMjRjOWViNzZiN2RkNzMwNmE3MDE2ZmI2NjY5","commit":{"author":{"name":"popcornmix","email":"popcornmix@gmail.com","date":"2018-10-23T15:19:02Z"},"committer":{"name":"popcornmix","email":"popcornmix@gmail.com","date":"2018-10-23T15:19:02Z"},"message":"kernel:

And rpi-update failing with *** Invalid hash given

popcornmix commented 6 years ago

Not what I see here:

pi@domnfs:~ $ curl -Ls https://api.github.com/repos/Hexxeh/rpi-firmware/commits/master | awk '{print $1}' |head
{
"sha":
"node_id":
"commit":
"author":
"name":
"email":
"date":
},
"committer":

curl -Ls https://api.github.com/repos/Hexxeh/rpi-firmware/commits/master | awk '{print $1}' |wc
   3387    3387   33912

(i.e. newlines between every line of json).

dbyio commented 6 years ago

Got it:

$ curl -q -Ls https://api.github.com/repos/Hexxeh/rpi-firmware/commits/master | awk '{print $1}' |head -2
{
"sha": 

My dot-curlrc breaking things

popcornmix commented 6 years ago

What was in your dot-curlrc?

dbyio commented 6 years ago

user-agent= masquerading as browser (IE).