SKS Menu Scrapper

Description

SKS Menu Scraper is a tool designed to automatically fetch and parse information about canteen menus, such as dish names, portion sizes, and prices. The project saves the scraped data into a database and provides a RESTful API for users to access menu items.

The SKS Menu Scraper also includes a feature that acts as a wrapper for external API to handle and track the number of canteen users - source.

Endpoints

The API is available at sks-api.topwr.solvro.pl. The following endpoints are available at the moment:

• GET /api/v1/meals
  • Description: Retrieves a list of all menu items, including dish names, sizes, and prices.
  • Note: If the SKS site does not display the menu, this endpoint returns an empty list.
  • Request:

    curl -X GET https://sks-api.topwr.solvro.pl/api/v1/meals

  • Response:

    [
    {
    "id": 85,
    "name": "Napój z soku jabłkowo-wiśniowego",
    "category": "DRINK",
    "size": "200ml",
    "price": "2.50",
    "createdAt": "2024-11-08T11:53:38.644+00:00",
    "updatedAt": "2024-11-08T11:53:38.644+00:00"
    },
    {
    "id": 84,
    "name": "Ziemniaki z koperkiem",
    "category": "SIDE_DISH",
    "size": "250g",
    "price": "4.50",
    "createdAt": "2024-11-08T11:53:38.644+00:00",
    "updatedAt": "2024-11-08T11:53:38.644+00:00"
    },
    {
    "id": 82,
    "name": "Pałki drobiowe w ciescie crazy",
    "category": "MEAT_DISH",
    "size": "250g",
    "price": "15.00",
    "createdAt": "2024-11-08T11:53:38.642+00:00",
    "updatedAt": "2024-11-08T11:53:38.642+00:00"
    }
    ]

• GET /api/v1/sks-users/current
  • Description: Retrieves the latest SKS user data with additional info about trend, whether the data is recent or not, and timestamp for the next scheduled update.
  • Request:

    curl -X GET https://sks-api.topwr.solvro.pl/api/v1/sks-users/current

  • Response:

    {
    "activeUsers": 1,
    "movingAverage21": 49,
    "externalTimestamp": "2024-11-11T13:40:00.000+00:00",
    "createdAt": "2024-11-10T23:00:00.116+00:00",
    "updatedAt": "2024-11-11T13:42:01.017+00:00",
    "trend": "STABLE",
    "isResultRecent": true,
    "nextUpdateTimestamp": "2024-11-11T13:47:31.017+00:00"
    }

• GET /api/v1/sks-users/today
  • Description: Retrieves the today's data about SKS users count
  • Request:

    curl -X GET https://sks-api.topwr.solvro.pl/api/v1/sks-users/today

  • Response:

    [
    {
    "activeUsers": 1,
    "movingAverage21": 49,
    "externalTimestamp": "2024-11-11T13:40:00.000+00:00",
    "createdAt": "2024-11-10T23:00:00.116+00:00",
    "updatedAt": "2024-11-11T13:42:01.017+00:00"     
    },
    "{...}"
    ]

Development

1. Clone the repository:

   git clone https://github.com/Solvro/backend-topwr-sks.git
   cd backend-topwr-sks

2. Install the required dependencies:

   npm install

3. Set up the PostgreSQL database:

   • Ensure PostgreSQL is installed and running.
   • Create a new database.
   • Update the .env file with your PostgreSQL credentials and database name.

4. Set up the environment variables in the .env file using the .env.example template.

5. Run migrations to create the database schema:

   node ace migration:run

6. Run scheduler for scrapper:

   node ace scheduler:run
   # or
   node ace scheduler:work

   Alternatively run scraping script once and individually:

   node ace scrape:menu
   node ace scrape:users

7. Start the development server:

   npm run dev

8. Access the data using:

   curl -X GET http://localhost:3333/api/v1/meals

Technologies

• Node.js
• Adonis.js
• PostgreSQL
• Coolify

Database Schema