Weather based crop advisory

[prtkjakhar]

Description

Buttons of different crops below weather widget which show a modal based advisory for that crop.

Need clarity on how can we show crops to farmers, different crops for each farmer based on region? API for this data? @ChakshuGautam @aashutosh-samagra @Amruth-Vamshi

[aashutosh-samagra]

• Blocked on KMAI PT - scraping data
• AKAI integration to be picked next week

[ChakshuGautam]

• [ ] @aashutosh-samagra to share the example doc in the desc
• [ ] @aashutosh-samagra to attach the figma to the desc
• [ ] @aashutosh-samagra to share the URL and frequency in the desc
• [ ] @rachitavya to scope and implement this similar to OUAT
• [ ] @prtkjakhar to update flows on the frontend and add telemetry
• [ ] AC
  • [ ] Similar URLs to OUAT
  • [ ] Frontend to match Figma
  • [ ] When a new advisory is published, it should show up on the screen

[aashutosh-samagra]

• IMD API documentation - https://mausam.imd.gov.in/imd_latest/contents/api.pdf
• Weather-based crop advisory to be scraped from - https://upcar.up.gov.in/en/page/advisory
• PRD and acceptance criteria - https://docs.google.com/document/d/1t2rH90ljuunX_MtNGFyO_m2NZ8l85bThxNCgvppScWc/edit#heading=h.xydwc6dekujv

[rachitavya]

Maintaining API automation scraping here: https://github.com/BharatSahAIyak/upcar-advisory

• [ ] Sanity check on the response we are getting in v1.
• [ ] Update docs for the APIs.

Response for this PDF: response.json

[KDwevedi]

PDFS

• Common/Unique fields in first 3 PDFs
• Discrepancies between current schema and PDF information
• Date, Weather datapoints, basic meaning of each heading i.e. what data is expected under that heading
• Data present/not present/misrepresented in schema
• parsing errors (not exhaustive is fine)
• Tests

[rachitavya]

Common/Unique fields in first 3 PDFs There is a same pattern in every PDF:

1. Introduction: No use for us
2. Next 1st week weather forcast
3. 8th day - 14th day weather forecast
4. General Advisory
5. Crop Specific advisories in bullets

6. References: No use for us

   Discrepancies between current schema and PDF information

Main discrepancies lies in weather_details: Previously: Daily weather data in numbers for parameters like min_temp, max_temp, min_rh, max_rh, wind_direction, cloud_cover, etc. Now in upcar we have: Two weeks data in paragraph having:

• Values of different regions for: min_temp - max_temp range and wind_speed
• Other values (not specific, can be random if any)
• Aggregate of weather conditions of different regions

parsing errors (not exhaustive is fine)

• Values for weather details are hallucinations of GPT. e.g. It is giving values for rh_max even if there is no mention of rh_max in the PDF.
• More detailed advisories in PDF, less advisories coming in response to us.

[KDwevedi]

Current Status:

• Agri advisory not fully represented in JSON form: Needs prompt engg
• Using IMDP weather APIs for weather information: /api/current_wx_api.php /api/cityweather.php

[KDwevedi]

@rachitavya

Get Accurate Data

• [ ] Weather APIs

• In these weather API, what information is coming, how is it mapped to location, demographic classification (district/block/city/station)

  • [ ] Match weather API data against UP districts
  • [ ] District-wise Data

• [ ] Crop Data

  • [ ] PDF Parsing
  • [ ] Issues:
    • [ ] Information is too concise / we are dropping relevant information
    • [ ] Probably fix: prompt engg/ parsed infomation chunking (headings are the same every time)

[rachitavya]

@prtkjakhar Can you elaborate on what basis we put the weather data on the bot. On the basis of location ? If yes, then how do we select the parameter from the apis. Do we select district or we need lattitude / longitude ?

I wanted to know about the flow so that we can figure out what to do for missing districts in imd APIs.

cc: @aashutosh-samagra @KDwevedi

[prtkjakhar]

@rachitavya if you are talking about the weather data we get inside message then I send lat. long. via socket. If you are talking about weather page then @Amruth-Vamshi has provided a bff api that takes lat. long. and returns weather data.

[rachitavya]

https://github.com/BharatSahAIyak/upcar-advisory

This repo is updated for:

• Complete automated pipeline to
  • scrap
  • ocr
  • translate
  • gpt call
• Getting advisories summarized without loosing crucial nouns.
• Will run every 10 days to ensure we don't miss any pdf which releases every fortnight.

Will share the API endpoint after deploying.

To do:

• [ ] Weather flow:
  • [ ] To figure out the stability of imd apis.
  • [ ] To get a clear transparency on how we will be accessing weather details.
  • [ ] Figure out what station code is.

@ChakshuGautam @aashutosh-samagra @KDwevedi

[KDwevedi]

API response might be unsuitable for weather:

• Data for all districts not available
  • /api/current_wx_api.php contains 10/75 districts
  • /api/cityweather.php contains 27/75 districts

Should re-evaluate weather pipeline

• [ ] validate frontend needs
• [ ] figure out alternate source for weather data @aashutosh-samagra

[KDwevedi]

blocked on weather APIs being down @aashutosh-samagra @ChakshuGautam

[aashutosh-samagra]

@rachitavya can you point me to the latest JSON which has the weather based crop advisory, i checked one JSON which had too concise information and relevant crops/lines were getting dropped.

[rachitavya]

@aashutosh-samagra https://github.com/BharatSahAIyak/upcar-advisory/blob/dev/latest.json

[aashutosh-samagra]

Weather APIs are now up @KDwevedi @rachitavya , we can schedule a call to discuss

[aashutosh-samagra]

@aashutosh-samagra https://github.com/BharatSahAIyak/upcar-advisory/blob/dev/latest.json

@rachitavya I checked this JSON only, we can relook at it since the information is too concise and relevant crops/lines are getting dropped.

[prtkjakhar]

Managing new tickets on the repo itself

[aashutosh-samagra]

Enhancements in UPCAR weather advisory API -

• [x] Crop advisory should be in points, summarized well and should include all nouns that are there in the actual data
• [ ] Nouns can be in the bold format (if possible)