Draft a microservice that pulls data from Kaggle

[tschaffter]

The idea is to develop a microservice that searches Kaggle competitions using its REST API and make these data available on OpenChallenges.

Workflow:

• Get a list a competitions from Kaggle using a set of categories (tags?)
• Add the new competitions to OpenChallenges DB, update the existing competitions

A variant of the above workflow would be to have 1) microservice that queries Kaggle and push the competitions to Kafka and 2) develop a microservice that listens to Kafka for incoming Kaggle competitions and stores the data to OpenChallenges DB.

The benefit of this approach is that querying Kaggle and processing the competitions received is separated, which provides more flexibility. For example, we could have more than one microservice that listen to Kafka to perform different actions.

Kaggle API

We can get information about Kaggle REST API via its Python client.

• Kaggle OpenAPI description
• See https://github.com/Kaggle/kaggle-api/search?q=competitions_list
• See https://github.com/Kaggle/kaggle-api/blob/49057db362903d158b1e71a43d888b981dd27159/kaggle/api/kaggle_api.py#L598-L612
• https://www.kaggle.com/general/63062#1042019

[tschaffter]

@vpchung Kaggle API enables to filter competitions by "categories". Could categories be the same thing as the tags that you used in the notebook?