This is just a suggestion for what we could use for API crawling, and by no means needs to me merged.
NodeRED is a really nice tool I've used for a few project before that lets you use node-based programming to string together events, and is really good for querying APIs. There's also super-lightweight versions available that run on RaspberryPIs without an issue!
Here's what I've configured so far:
The above only runs the API requests when triggered with the button (grey, top left), and then dumps them to the internal log (green, bottom right). We can configure it to start flows when all sorts of things happen, including requests, timers, even text message events.
Some of the nodes run custom javascript functions, which can act as filters or perform other actions on messages: in the above, they add in environment variables for the requests, and then strip them back out again for the logs. However we can use them to do all sorts of things, even launch other programs or Python scripts!
What's also really nice is, with the plugins, NodeRED speaks just about every dialect of database, so we can easily use it to clean, partition and then pass the data to the database.
I suggest we use this tool since it's really easy to gain a quick overview of our data pipelines with the graphical interface, and makes prototyping new APIs very easy :)
This is just a suggestion for what we could use for API crawling, and by no means needs to me merged.
NodeRED is a really nice tool I've used for a few project before that lets you use node-based programming to string together events, and is really good for querying APIs. There's also super-lightweight versions available that run on RaspberryPIs without an issue!
Here's what I've configured so far:
The above only runs the API requests when triggered with the button (grey, top left), and then dumps them to the internal log (green, bottom right). We can configure it to start flows when all sorts of things happen, including requests, timers, even text message events.
Some of the nodes run custom javascript functions, which can act as filters or perform other actions on messages: in the above, they add in environment variables for the requests, and then strip them back out again for the logs. However we can use them to do all sorts of things, even launch other programs or Python scripts!
What's also really nice is, with the plugins, NodeRED speaks just about every dialect of database, so we can easily use it to clean, partition and then pass the data to the database.
I suggest we use this tool since it's really easy to gain a quick overview of our data pipelines with the graphical interface, and makes prototyping new APIs very easy :)