search

ChakshuGautam / geoquery.in

Service to get Location from ipv4 and ipv6
https://geoquery.in
29 stars 76 forks source link

[DMP 2024]: Design Map Data Storage Backend #125

Open ChakshuGautam opened 6 months ago

ChakshuGautam commented 6 months ago

Description

GeoQuery.in is aims to become India's leading free and open-source mapping tool, dedicated to offering precise and current government body geographical information accessible to everyone. Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. However, as the number of features in the map increases with scale, we need to redesign the backend to support the storage of Map Data as well as data for layers above it + features. This involves a total redesign of the backend and will be broken down into several steps.

Currently, we are storing JSON Data everywhere. With scale, the number of features in the map would increase

The objective is to redesign the backend to support for storage of Map Data as as well as data for layers above it + features. This is a total redesign of the backend.

Goals & Mid-Point Milestone

Setup/Installation

See here.

Expected Outcome

A redesigned backend system for GeoQuery.in capable of efficiently storing Map Data, layers above it, and features, ensuring scalability and backward compatibility.

Acceptance Criteria

Implementation Details

Mockups/Wireframes

No response

Product Name

GeoQuery.in

Organisation Name

SamagraX

Domain

⁠Agriculture

Tech Skills Needed

JavaScript

Mentor(s)

@ChakshuGautam

Category

Backend, Other

2008sahil commented 4 months ago

I am writing to express my keen interest in contributing to the GeoQuery.in project as outlined in the project description. After reviewing the project requirements, I believe that MongoDB could be a suitable choice for the backend database to achieve high scalability and performance.

Having worked on several projects utilizing MongoDB, I am confident in its ability to handle diverse types of data efficiently, especially geographical data for a mapping tool like GeoQuery.in. MongoDB's flexibility, geospatial querying capabilities, scalability, and integration with Node.js align well with the project's objectives.

Before proceeding further, I would like to propose MongoDB as the database solution for the project. I believe it can offer the performance and scalability required to support the storage of map data, layers, and features, while ensuring efficient retrieval and manipulation of geographical information.

However, to ensure alignment with the project goals and requirements, I would appreciate confirmation from our mentor, @ChakshuGautam, regarding the suitability of MongoDB for the GeoQuery.in backend. I am eager to discuss this proposal further and collaborate with the team to make informed decisions that contribute to the success of the project.

Looking forward to your feedback and guidance on this matter.

AbhimanyuSamagra commented 4 months ago

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

Riyasharma28 commented 4 months ago

Good evening Respected Mentor @ChakshuGautam

I'm a Full Stack Developer proficient in HTML, CSS, JavaScript, MongoDB, React.js, Node.js, Bootstrap, Database Management, UI/UX Design. With over 10 projects completed in similar domains, I'm eager to contribute my expertise to the project. I'm excited about the opportunity to collaborate and drive its success. Looking forward to discussing further.

Anushkabh commented 4 months ago

@ChakshuGautam i am facing difficulty in setting up the project locally .. the script file is not running i followed the instruction still . Kindly guide

swastik-akhil commented 4 months ago

@Anushkabh what issue you are encountering?

Anushkabh commented 4 months ago

@swastik-akhil basically the setup.sh file is not running hence bun not installed

swastik-akhil commented 4 months ago

What error it is showing?

Anushkabh commented 4 months ago

Screenshot (131) bun : The term 'bun' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that
the path is correct and try again. At line:1 char:1

swastik-akhil commented 4 months ago

Actually the bun is not present in your local machine that's why this error is occuring. Go to bun website and then install it from there. You can also follow some tutorial and go along with it. Also make sure to restart your system after the installation.

Anushkabh commented 4 months ago

Screenshot 2024-04-25 120927 now i am encountering this @ChakshuGautam @swastik-akhil

swastik-akhil commented 4 months ago

@ChakshuGautam I am also encountering the same issue. The file PARSED_MASTER_LOCATION_NAMES.json is not present which is causing the issue.

Anushkabh commented 4 months ago

@ChakshuGautam kindly guide further

CodrAyush commented 4 months ago

@ChakshuGautam uptill what level of javascript one should know to contribute to this project...

Savio629 commented 4 months ago

@Anushkabh In the mean time, you can run it on github codespace. I have tested it out. Also @swastik-akhil if you were facing some issues on codespace try running it on gitpod

Ishu2412 commented 4 months ago

hey i was installing the project and getting this error

image
Savio629 commented 4 months ago

@Ishu2412 Please share OS of your system and also which commands have you tried to run? It's difficult to debug by just seeing the image

Anushkabh commented 4 months ago

@Savio629 Thankyou , its running now Screenshot 2024-04-26 233750

swastik-akhil commented 4 months ago

@ChakshuGautam I am also encountering the same issue. The file PARSED_MASTER_LOCATION_NAMES.json is not present which is causing the issue.

@Savio629 @ChakshuGautam Kindly help me with this one.

Anushkabh commented 4 months ago

@Savio629 are there any existing issues which i can solve and include in my proposal?

Ishu2412 commented 4 months ago

@Ishu2412 Please share OS of your system and also which commands have you tried to run? It's difficult to debug by just seeing the image

I am using windows 10 and also I followed the command instruction from the setup link which you have provided

35C4n0r commented 4 months ago

@ChakshuGautam @Savio629 @dhruv-1001 Just a few questions issue: https://github.com/ChakshuGautam/geoquery.in/issues/125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis.. Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage. Or the preference leans towards a NoSQL database

ChakshuGautam commented 4 months ago

@ChakshuGautam @Savio629 @dhruv-1001 Just a few questions issue: #125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis.. Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage. Or the preference leans towards a NoSQL database

Although the project's goals are not to scrape - but to build APIs that would help us to store the information. Information gathering is a separate task that we should create a ticket for. Similar exercise was done for Districts and Villages earlier.

Would you (or anyone else) like to take a crack at creating a ticket? I can review and we can get started with collecting the data so that by the time DMP is completed we have a repository of all Govt related buildings too.

ChakshuGautam commented 4 months ago

@Ishu2412 Please share OS of your system and also which commands have you tried to run? It's difficult to debug by just seeing the image

I am using windows 10 and also I followed the command instruction from the setup link which you have provided

Can you create a separate ticket for this? We would want to support windows long term but currently prioritizing linux. Please get started with WSL. Thanks.

I am closing this comment.

ChakshuGautam commented 4 months ago

@Anushkabh would you like to raise a PR for documentation? I believe we are lacking on that front and your inputs would be valuable.

@Savio629 can you create a new ticket for Documentation and direct everyone there? Thanks.

Anushkabh commented 4 months ago

@Anushkabh would you like to raise a PR for documentation? I believe we are lacking on that front and your inputs would be valuable.

@Savio629 can you create a new ticket for Documentation and direct everyone there? Thanks.

Okay i'll do that

vivekmishra9631 commented 4 months ago

Hi @ChakshuGautam , I'm Vivek Mishra, a third-year NIT Karnataka student with a strong passion for building impactful web applications. I'm excited to express my interest in the Geoquery project!

My background includes:

Full-Stack Development Experience: I honed my skills as a full-stack developer intern at aiRender for 6 months, working with JavaScript, ReactJS, Redux, and WebRTC. This experience solidified my understanding of both front-end and back-end development.

MERN Stack Expertise: In addition to my internship, I'm also a proficient MERN stack developer, comfortable working with MongoDB, Express, React, and Node.js. This versatility allows me to tackle projects with both front-end and back-end components.

My problem-solving skills are further bolstered by my achievement of a gold medal at Worldquants Brain, demonstrating my ability to think critically and find efficient solutions.

I believe my technical skills, combined with my enthusiasm for building meaningful projects, make me a strong candidate to contribute to the project. I'm eager to learn more about the specific requirements and how my expertise can be utilized.

DGRYZER commented 4 months ago

Hello, My name is Debajyoti Ghosh. I am a Frontend Developer (Fresher). I am sharing some features that this should have. These features are following -

  1. Requirement Analysis and Planning:

    • Gather detailed requirements from stakeholders regarding the desired functionalities and performance expectations.
    • Plan the project timeline, resource allocation, and milestones for each phase of the redesign.

  2. Backend Redesign Strategy:

    • Define a clear strategy for redesigning the backend to support storage for map data, layers, and features.
    • Determine the technology stack and architecture best suited for the project's requirements, considering scalability, performance, and maintainability.

  3. Data Modeling and Database Design:

    • Design the database schema to efficiently store map data, layers, and features.
    • Choose appropriate database technologies and optimize data structures for fast retrieval and scalability.

  4. API Development:

    • Develop RESTful APIs for CRUD operations on map data, layers, and features.
    • Implement authentication and authorization mechanisms to secure the APIs and control access to sensitive data.

  5. Real-time Data Processing:

    • Implement real-time data processing capabilities to ensure timely updates and responsiveness.
    • Utilize technologies like WebSockets or server-sent events for real-time communication between the backend and frontend.

  6. Scalability and Performance Optimization:

    • Optimize database queries and API endpoints for performance and scalability.
    • Implement caching mechanisms to reduce latency and improve overall system responsiveness.

  7. Documentation and Testing:

    • Create comprehensive documentation for APIs, data models, and system architecture.
    • Write unit tests, integration tests, and end-to-end tests to ensure the reliability and functionality of the backend components.

  8. Deployment and Monitoring:

    • Deploy the redesigned backend to production servers or cloud platforms.
    • Set up monitoring tools to track system metrics, identify performance bottlenecks, and troubleshoot issues in real-time.

  9. User Training and Support:

    • Provide training sessions for users and administrators on using the new backend features and functionalities.
    • Offer ongoing support and maintenance to address any issues or enhancements post-deployment.

  10. Feedback Collection and Iteration:

    • Gather feedback from users and stakeholders to evaluate the effectiveness of the redesigned backend.
    • Iterate on the design and implementation based on feedback to continuously improve the system.

Thank You. Debajyoti Ghosh.

dev-chouhan commented 4 months ago

Good day, @ChakshuGautam. My name is Devraditya Singh Chouhan, and I'm a third-year student at UEMJ who is really passionate about creating powerful web applications. I'm thrilled to show my support for the Geoquery project!

My history consists of:

Experience with Full-Stack Development: I spent two months working with JavaScript, HTML, CSS, Tailwind, and SQL as a full-stack developer intern at Zonion, where I refined my skills. My grasp of both front-end and back-end development has been strengthened by this experience.

MERN Stack Expertise: In addition to my internship, I have experience with MongoDB, Express, React, and Node.js as a competent MERN stack developer. I can work on projects including both front-end and back-end components because of my versatility.

I think my passion for creating worthwhile projects and my technical expertise make me an excellent choice to contribute to the project. I'm excited to find out more about the precise specifications and how my experience can be put to use.

Mikhiel39 commented 4 months ago

Hello, My name is Mikhiel Sujeet Benji, and I'm a full-stack developer who is interested in your project and would like to contribute. I have a few suggestions for features that could be added to this project: 1) Custom Map Styling: Users should be able to customize the appearance of their maps by adjusting colors, styles, and labels to suit their preferences. 2) Offline Map Support: Implementing offline map support would allow users to access map data and features even in areas with limited or no internet connectivity. 3) Accessibility Features: It would be great to ensure accessibility for users with disabilities by implementing features such as screen reader support, keyboard navigation, and high-contrast mode.

35C4n0r commented 4 months ago

@ChakshuGautam @Savio629 @dhruv-1001 Just a few questions issue: #125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis.. Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage. Or the preference leans towards a NoSQL database

  • The preference is to use PostGIS.
  • Mandis data can be found here - https://enam.gov.in/web/apmc-contact-details. We will have to scrape for it.
  • For other Govt buildings - we would need to figure out similar sources. One that I have found is https://igod.gov.in/, which includes sectors at the bottom which has a drill down. We will have to figure out how to scrape those as well.

Although the project's goals are not to scrape - but to build APIs that would help us to store the information. Information gathering is a separate task that we should create a ticket for. Similar exercise was done for Districts and Villages earlier.

Would you (or anyone else) like to take a crack at creating a ticket? I can review and we can get started with collecting the data so that by the time DMP is completed we have a repository of all Govt related buildings too.

@ChakshuGautam I've sent you the proposal for feedback, if you find some time available, kindly take a look at it.

li-veindark commented 4 months ago

Hello @ChakshuGautam. Can you explain me what do you mean by "layers above the data"? Are these layers are not same as the layers in the "openStreetMap"? If not, how?