lgirdk / boardfarm

Automated testing with python
BSD 3-Clause Clear License
21 stars 33 forks source link

Add global locking support #296

Closed mattsm closed 5 years ago

mattsm commented 5 years ago

The idea is a device could be used in two places for two boards so we want to globally lock it. Here are a few ideas:

https://pypi.org/project/python-redis-lock/ ^ This looks simple, run a redis server, point at it from boardfarm config, and implement simple device locking off a key value in the boardfarm config?

Other possiblities: http://python-rq.org/ https://www.rabbitmq.com/ http://www.celeryproject.org/ https://github.com/binarydud/pyres https://github.com/coleifer/huey

mbanders commented 5 years ago

Instead of having the boardfarm config just be a JSON file, we could setup a server that generates the JSON data live as users request it. Each user would be shown only the relevant config parameters for the amount and type of devices that they request. And we could enforce limits like if a device only supports one user, then we won't show that device to anyone else until it is not being used.

Lets say the server is boardfarm.whatever.com. Then when a user runs bft it could do an http POST and include info about what it wants to check out. In the form of a curl command as an example:

curl -X POST \
     -H 'Content-Type: application/json' \
     -d '<filter>' \
     http://boardfarm.whatever.com/api/checkout

Where <filter> above would be JSON-formatted data that specifies what the user needs to check out. Then the server responds with JSON-formatted data that looks like the boardfarm config file you have today - except the server filtered to just the one station and devices that you need.

Behind the scenes the server can mark devices as "inUse: true" or it could increment a counter like "activeUsers: 1". And if a device reaches the max number of users, then nobody else can check it out.

When bft finishes it could POST to /api/checkin with info so the server can mark those devices as no longer in use. And now someone else can check them out.

The other big advantages to doing this:

In the case that bft crashes and fails to checkin devices, the server could just have a timeout. Like after an hour, it will mark those devices as not in use (or some reasonable amount of time).

mbanders commented 5 years ago

Also if we have a database we can easily then have a pretty front-end for people to look at to see what currently is in use, what's available, what's been run...

Screen Shot 2019-06-13 at 5 03 22 PM

This isn't Kibana/Elastic Search by the way, this would be a live front-end. I've made a few like these.

mattsm commented 5 years ago

few comments:

  1. make sure we have a keepalive so if bft does not check the boards back in they become available again, would be nice to check that it's free and/or kick the user off as well.
  2. we have feature flags and filters too, so be sure to pass those to the server to pick an appropriate board
  3. if the user still points at a json, it would be nice to use the same code just on a single machine (for testing purposes if nothing else) we can also move some stuff from bft to it's own file or library.
  4. for the visualizations, it would be nice to pass more details about the user, ip address, etc... eventually we could do authentication as well.
mbanders commented 5 years ago

This is done (though still an on-going project) now here: