Shruti3004
commented
1 year ago Hey @fatchat can you please also add Project size and milestones
Closed fatchat closed 1 year ago
Shruti3004
commented
1 year ago Hey @fatchat can you please also add Project size and milestones
fatchat
commented
1 year ago @siddhant3030 could you please resolve Shruti's comment
siddhant3030
commented
1 year ago Hey @fatchat can you please also add Project size and milestones
Hey. The project size is Large and I have added a few milestones which can help you focus. Let me know if you want to discuss more
Siddhant2021
commented
1 year ago hi @siddhant3030 , i am new to open source and a bit lost after the project setup part. Can you brief a little on what we have to do (that's the acceptance criteria part i don't understand much).
siddhant3030
commented
1 year ago @Shruti3004 @Siddhant2021 are you guys on discord? We can help you there and connect with you guys if you need help. here's the invite link https://discord.gg/5etKnjPx . Do join us
Shanzal04
commented
1 year ago Hey @siddhant3030 , can u please elaborate the milestone a bit more.
siddhant3030
commented
1 year ago @Shanzal04 I have added a few details. Please look into that and let me know if you didn't understand something.
SigireddyBalasai
commented
1 year ago I am looking forward to contribute
FirePing32
commented
1 year ago Hey @Shruti3004 @fatchat This seems like an interesting project. I have worked on something similar before and am eager to contribute to this project. Looking forward to submitting a proposal soon 🚀
Darshan2003
commented
1 year ago @Shruti3004 @Siddhant2021 are you guys on discord? We can help you there and connect with you guys if you need help. here's the invite link https://discord.gg/5etKnjPx . Do join us
Hey @siddhant3030, the invite link seems to be expired can you share the new link?
Intro
Airbyte is an open-source ELT tool with a large ecosystem of connectors to various data sources and destinations. They encourage the development of connectors by third-parties, and their platform is designed to work with connectors adhering to their specification.
Avni is a data-collection platform used widely in the development sector. They allow the creation of complex forms and workflows for field workers, with features like multi-language support and offline-mode.
API Guide for AVNI
https://avni.readme.io/docs/api-guide
The Task
We would like to build a source connector for Avni. Doing this will allow Avni's NGO users to leverage Airbyte for their data management needs.
Airbyte connectors can be written in Java or Python. The Tech4Dev team has experience in building connectors for other development sector data-collection tools, for example:
We will guide you through the design, development and eventual submission to Airbyte.
You will learn how to contribute to an open-source project, including
If you want to simultaneously contribute to an open-source project and to the development sector, this is a great project for you.
Features to be implemented
Learning Path
Project setup
Read this and clone this repository
Acceptance Criteria
Milestones
Understand the Platform for Avni and create a sandbox account:
In this milestone, you need to familiarize yourself with the Avni platform. Start by creating a sandbox account on Avni. A sandbox account provides a safe environment for development and testing without affecting production data.
Explore the APIs of Avni :
Once you have your sandbox account, explore the Avni APIs documentation. Understand the available endpoints, request methods (such as GET, POST, PUT, DELETE), and request/response formats (such as JSON, XML).
Implement the basic read of the connector:
Choose a specific connector you want to work with and implement the basic read functionality. This means retrieving data from the connector using the Avni API. This could involve making a GET request to the appropriate endpoint with the required parameters, such as specifying the data source and any filters. Process the API response and extract the relevant data that you need.
Check the API success while making a connection and throw some errors if the credentials are wrong:
When making API requests to Avni, it's essential to handle potential errors effectively. Implement a mechanism to check the success of the API connection before proceeding further. This can involve checking the response status code, such as a 200 OK indicating a successful connection. Additionally, if the authentication credentials are incorrect, handle the error gracefully by providing appropriate feedback or error messages to the user.
Pull data from a particular date:
Extend your read functionality to pull data from a specific date or date range. You may need to modify your API request parameters to include the desired date or date range as filters. This allows you to retrieve only the data that has been updated or added since the specified date, improving efficiency when synchronizing data.
Implement incremental sync:
Incremental sync is the process of synchronizing only the changes or updates since the last synchronization instead of retrieving all the data again. Implement this functionality by keeping track of the last synchronization timestamp or a synchronization token. Use this information to fetch only the modified or new data from Avni since the last synchronization. Update your local system accordingly to reflect the changes.
Write tests
Create test cases to ensure the functionality of your connector. Use Airbyte's provided testing utilities and examples to guide you.
Document your connector
Document your connector to provide clear instructions on how to use it. This includes documenting the connector's configuration options, any specific considerations or limitations, and examples of usage. Follow Airbyte's documentation guidelines to ensure consistency.
How to contribute
Familiarize yourself with the project
Start by exploring the Airbyte project on GitHub. Visit the project repository and read through the documentation, including the README file, contributing guidelines, and code of conduct. Understand the project's goals, architecture, and the current state of development.
Set up the development environment
Follow the instructions provided in the Airbyte repository to set up the development environment. This typically involves installing the necessary dependencies, setting up a local development instance, and running tests to ensure everything is working correctly.
Fork the repository
Fork the Airbyte repository on GitHub to create your own copy of the project. This will allow you to make changes and submit pull requests.
Create a new branch
Create a new branch in your forked repository to work on your contribution. It's recommended to create a branch with a descriptive name that reflects the nature of your changes.
Make the necessary changes
Write code, fix bugs, or make the desired improvements in your branch. Follow the project's coding standards and conventions. Make sure to write clear and concise commit messages for each change.
Test your changes
Ensure that your changes work as expected by running tests provided by the project. This helps maintain the quality and stability of the codebase.
Submit a pull request
Once you are confident with your changes, submit a pull request (PR) to the main Airbyte repository. Provide a detailed description of your changes, explaining the problem you addressed or the feature you added. Include any relevant documentation updates and tests.
Collaborate and iterate
Engage in discussions with the Airbyte community and maintainers regarding your contribution. Address any feedback or code review comments promptly. Iterate on your changes as necessary until they are ready to be merged into the main codebase.