JSON API for collaborative area scanning challenge

takluyver commented 6 years ago

We have proposed for 2018 to add a collaborative element to the area scanning challenge: there will be a large grid, with the points for each square divided among the boats that have entered it, making it most valuable to visit squares which no other boat has visited.

The simplest way to allow coordination without requiring specific hardware for the boats is to expose the real-time tracking data as a JSON API. This will provide:

[x] The current position of all active boats
- [ ] Documented?
[ ] For each grid square, how many boats have already visited it (and maybe which boats?)
[ ] Stretch goal: allow boats to share data on where they plan to go next?

It will be up to the teams how to get this data to their boat - e.g. whether it has a mobile data modem to query it directly, or a laptop queries it and sends the data to the boat in another form. (Or they can ignore the API)

kirs commented 6 years ago

Do you think these three data sources should be different HTTP endpoints?

I think that at least "The current position of all active boats" should be its own endpoint, because it might be too expensive to calculate each grid square if you only want to get current positions of boats.

takluyver commented 6 years ago

Yep, I think it probably makes sense to do them as separate endpoints.

takluyver commented 6 years ago

I've been reminded that we should try to get this defined, because teams will need to know the interface if they're going to code against it. So, I propose:

Current boat positions

[
  {
    "name": "Black Python",
    "boat_id": 3,
    "latitude": 50.909698,
    "longitude": -1.404351,
    "updated": "2018-04-29T18:31:05Z"
  }
]

An array of objects, one per active boat.

The timestamp should be in the standard ISO 8601 format, which might not be exactly what I put above, and in UTC.

Visited squares

[
  {
    "axis1": 0,
    "axis2": 0,
    "visited_by_boats": [3, 5],
  }
]

I'm thinking that we define our own coordinate system for the grid, so each square has integer coordinates. This structure allows some squares to be missing, so the grid doesn't have to be a complete rectangle, but it does assume that all squares are the same size and alignment. @smaria does that fit with what you're thinking? Or do you want to allow for more complex search areas with multiple grids?

If we want more flexibility, we could specify lat/long for 2 corners of each (assuming cells are square), 3 corners (rectangles), or specify all the corners so cells can be arbitrary polygons. Hexagonal grid, anyone? :-)

This API (however we define it) also lends itself to building some nice visualisations where cells get coloured according to how many boats visited them.

kirs commented 6 years ago

@takluyver should those endpoints be scoped either by a mission, a team, a race edition?

takluyver commented 6 years ago

By a mission, I think. This will only be used for one of the four missions, and the grid-cells part only makes sense for that mission, because the others don't define a grid. It should be possible to retrieve the status with respect to all boats/teams in a single API call.

(I may use the terms boats and teams interchangeably. I know the code allows for more than one boat per team, but in practice almost all teams have exactly one boat. Feel free to check if it's not clear what I mean)

smaria commented 6 years ago

in practice almost all teams have exactly one boat As per rules, line 27: "Each team competes with one boat; the team members can be shared among different teams." At least for now, the words 'team' and 'boat' can be treated as synonyms.

The timestamp should be in the standard ISO 8601 format, which might not be exactly what I put above, and in UTC.

I first thought of using the CSV-2s format in the rules (hhmmssdd), but the suggested ISO 8601 makes more sense to me.

I'm thinking that we define our own coordinate system for the grid, so each square has integer coordinates. This structure allows some squares to be missing, so the grid doesn't have to be a complete rectangle, but it does assume that all squares are the same size and alignment.

For this year one grid will definitely suffice. Whilst it probably will be L-shaped to maximise the area, this should already be taken care of by the empty squares.

In the future we may want to use several separate grids, at different resolutions, since this represents real applications better. From what I have seen, in practice this would be handled by different grid IDs with corner points/resolution information. For this it may be worth adding a third identifier for grid ID, though alternative ideas of handling the whole area scanning task may emerge by that time. I will add this to my discussion notes for the conference.

takluyver commented 6 years ago

I'm now wondering if it's actually easier to define grid cells with lat/long of corners, rather than our own coordinate system. It means some redundant information, but there's less external information needed to make sense of it.

smaria commented 6 years ago

So the suggested format would be something like this?

[
  {
    "lat_min": 0,
    "lat_max": 0,
    "lon_min": 0,
    "lon_max": 0,
    "visited_by_boats": [3, 5],
  }
]

This might be easier to work with for teams as well, and it definitely has advantages for archiving.

takluyver commented 6 years ago

Min/max only works for axis-aligned rectangles, where the sides are precisely NESW. I'd probably go for something like the way GeoJSON defines polygons, which is a list of long-lat pairs:

[
  {
    "coordinates": [[
            [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],
            [100.0, 1.0], [100.0, 0.0]
          ]],
    "visited_by_boats": [3, 5],
  }
]

In fact, we could make the entire response GeoJSON, with the "visited_by_boats" information as a 'property' attached to each polygon. That's probably nicer than defining a format ourselves.

takluyver commented 6 years ago

I've updated the original issue body with a checklist. We should also make sure that we document any APIs we provide that we expect teams to use.

takluyver commented 6 years ago

@kirs thanks for the current positions API - are you still interested in working on a grid-cell visitation API as well? I think this could return GeoJSON polygons.

takluyver commented 6 years ago

I now think that the 'which boats have visited each square' API should return GeoJSON, something like this:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],
            [100.0, 1.0], [100.0, 0.0]
          ]
        ]
      },
      "properties": {
        "visited_by_robots": [1046216088, 995256502]
      }
    },
    ...
  ]
}

i.e. each cell in the grid is a single polygon, with corners defined by longitude-latitude pairs (the first and last are the same, to close the loop), and the visited_by_robots property is a list of robot IDs.

takluyver commented 6 years ago

Is anyone who's familiar with Ruby likely to have some time to build this API before the competition at the end of this month? Do you know anyone else who might be interested in helping with this?

If not, the backup plans are:

Write a separate server (in Python, which I'm familiar with), which polls the position API that @kirs added (thanks!), and serves the squares-visited API. Not ideal because then we have two separate servers to host and maintain.
Only offer the position API, and let teams track which boat has visited which square by themselves. Also not ideal, because the squares-visited API would be by far the easier one for teams to consume.

Thanks everyone for your help - not long now until the competition! :boat: :sun_with_face:

chinchifou commented 6 years ago

Hello,

Unfortuanltely I do not have much time to offer and no longer have a Rails environnement on my computer and will not be able to test anything ...

However if you are okay with a "non perfect but easy Ruby solution" and are able to do all the testing part I can give you some code samples to help you with.

Both solutions assumed that the areas are rectangles defined by two coordinates.

Reminders (if the database did not changed) :

A "mission" contains multiple "attempts"
A "robot" has multiple "attempts" for a "mission"
"tracker-id" and "robot_id" are linked together for each "attempt" (so that you can change the tracker of a robot during the competition)
A "coordinates" only contains "tracker_id"

SOLUTION 1 - easy but dirty as it almost a copy-paste of the existing method "gatherCoordsBetweenDates": limitations :

above all, does not filter the coordinates by mission and will put a lot of stress on the server
does not respect the sugggested JSON API (it just returns a list of tracker_id with ducplicates)
does not respect the ISO format for the date
return a tracker_id instead of a robot_id
not tested but :
fast to implement without too much risks

in /config/routes.db : get 'gatherCoordsBetweenCoords', to: 'coordinates#gatherCoordsBetweenCoords

in /app/controllers/coordinates_controller.rb :

def gatherCoordsBetweenCoords
  if (params[:firstlat] != nil && params[:lastlat] != nil && params[:firstlong] != nil && params[:lastlong] != nil)
    if (first_lat < last_lat && first_long < last_long)
      first_lat = params[:firstlat]
      last_lat = params[:lastlat]
      first_long = params[:firstlong]
      last_long = params[:lastlong]

      needed_coords = Coordinate.where("? < latitude AND latitude < ?", first_lat, last_lat).where("? < longitude AND longitude < ?", first_long, last_long).order(datetime: :desc).order(tracker_id: :asc)

      #TODO improvement : concatenate based on tracker_id + remove duplicates ?
      render json: needed_coords.to_json(:only =>[:tracker_id])
    else
      render plain: "?invalid coordinates, make sure that the given ccordinates represent a rectangle", status: 400
    end
  else
    render plain: "?invalid coordinates, some coordinates are missing", status: 400
  end
end

It should works like this : Input HTML GET request : GET /coordinates/gatherCoordsBetweenCoords?firstlat=1234&lastlat=1234&firstlong=1234&lastlong=1234 JSON response :

{
  {
    "tracker_id":1,
  }, 
  {
    "tracker_id"=1,
  }, 
  {
    "tracker_id"=2,
  }
}

SOLUTION 2 - more complex and definitively not tested (based on "gatherCoordsBetweenDates" and "latest_by_mission" methods) : limitations :

does not respect the sugggested JSON API
return duplicates (but less than solution 1)
clearly not tested but :
above all, filter based on mission (less stress on the server)
respect the format for the date
less dirty

in /config/routes.db : get 'gatherCoordsBetweenCoords', to: 'coordinates#gatherCoordsBetweenCoords

in /app/controllers/coordinates_controller.rb :

def gatherCoordsBetweenCoords

  mission = Mission.find(params[:id])

  if mission != nil
    if (params[:firstlat] != nil && params[:lastlat] != nil && params[:firstlong] != nil && params[:lastlong] != nil)
      if (first_lat < last_lat && first_long < last_long)

        first_lat = params[:firstlat]
        last_lat = params[:lastlat]
        first_long = params[:firstlong]
        last_long = params[:lastlong]

        robots_having_visited_the_area = mission.attempts.map do |attempt|

          coordinates = attempt.coordinates.order(datetime: :desc).where("? < latitude AND latitude < ?", first_lat, last_lat).where("? < longitude AND longitude < ?", first_long, last_long).map do |coordinate|
            coordinate.as_json(only: [:latitude, :longitude]).merge(datetime: coordinate.datetime_as_time.iso8601)
          end

          {
            robot_id: attempt.robot.id
          }
        end

        render json: robots_having_visited_the_area

      else
        render plain: "?invalid coordinates, make sure that the given ccordinates represent a rectangle", status: 400
      end
    else
      render plain: "?invalid coordinates, some coordinates are missing", status: 400
    end
  end
end

It should works like this : Input HTML GET request : GET /coordinates/gatherCoordsBetweenCoords?mission_id=1&firstlat=1234&lastlat=1234&firstlong=1234&lastlong=1234 JSON response :

{
  {
    "robot_id":1,
  }, 
  {
    "robot_id"=1,
  },
  {
    "robot_id"=2,
  },
}

It is way from being perfect but I hope it may help you anyhow ;)

I'm happy to help if needed. Good luck!

WRSC / tracking

JSON API for collaborative area scanning challenge #3

Current boat positions

Visited squares