Designing User Entity and Persistence of user data

[parthasarathydNU]

Overview

As part of the development of the AutoGrader project, a robust and scalable database schema is essential to support the dynamic features detailed in the UI design (Issue #7). The goal is to create a database schema that not only accommodates the current functionalities but is also flexible enough to handle future expansions.

Repo:

https://github.com/autograder-org/autoGrader-api

Objectives

• Schema Creation: Develop a comprehensive database schema that integrates user profiles, rubric styles, and saved rubrics.
• Support for UI Features: Ensure the schema supports all features outlined in the UI design, including assignment creation, rubric management, and persona interaction.
• Future-Proof Design: Prepare the schema for upcoming features and potential enhancements.

Required Features Integration

The schema must support:

• Assignment Creation: Ability to store and retrieve assignments created via the chat interface.
• Rubric Creation: Storage of user-generated rubrics and their associated parameters.
• Workspace: Organizational tools for users to manage their historical data such as past chats, assignments, and rubrics.
• Persona Tab: Storage of persona details including expertise, grading style, animated profiles, and voice settings.

Upcoming Features Support

Prepare the database for:

• Conversation Continuation: Storing session data to allow users to pick up where they left off.
• Persona Management: Flexibility to add new personas or modify existing ones.
• User Profile Customization: Extended user profile settings, including voice preferences and persona-specific configurations.

Design Considerations

• Scalability: Design a schema that can scale efficiently as the number of users and data volume grows.
• Security: Incorporate security measures to protect user data, especially personal identifiers and custom settings.
• Performance: Optimize the schema for quick queries, especially for operations related to frequent and critical features such as assignment grading and rubric application.

Feedback and Iteration

• Collaborative Review: Engage with both the backend and frontend teams to ensure the schema meets all technical and user-experience requirements.
• Iterative Development: Based on initial reviews and user feedback, iteratively refine the schema to address practical issues and incorporate new requirements as the project evolves.

Documentation

• Schema Documentation: Provide detailed documentation of the database schema, including entity-relationship diagrams and descriptions of each entity and relationship.
• Change Log: Maintain a record of all changes to the schema, including the rationale behind each modification.

[lokvenkatesh]

Designing User Entity and Persistence of user data.docx

[parthasarathydNU]

Designing User Entity and Persistence of user data.docx

@lokvenkatesh - the document looks good, has covered most of the aspects required for the feature implementation. I noticed that you have gone with a Relational Database implementation with Normalized data. I would like to know the rationale behind it and why not go with a NoSQL database. I want to evaluate that aspect before we start implementing this as part of an API.

[techdevsynergy]

@parthasarathydNU the main thing you need to think conceptually in deciding sql or nosql is if your application need select, update, delete and insert / multiple users will access the same record then you need to go with sql (RDBMS) since its ACID compliant. Most of the nosql DB's are ACID non compliant except mongodb meaning only if your application has select,insert and delete you need to roll with it if multiple user update the same record it might not update the record properly in right sequence. For your use case its better to go with rdbms since i dont think you will get terabytes of data like logs storage where nosql is really good option. Hope this helps in your decision :)

[parthasarathydNU]

@dharsan18 In this case, we will be isolating data per user to start off with, we don't have the concept of a team / multiple users working on the same entity yet. Later we might include features to allow students and professors to to access shared resources. The first version, would just include a user logging in, opening the app and trying out the different features - Assignment Creation, Rubric Creation and a coming soon card that talks about subsequent features. I kind of like the way you are justifying the use of RDBMS here. Further I think that given the fact all the views on the front end for this feature need normalized data ( user profile just needs user data ), Persona Tab just needs persona data, we can stick to RDBs for now. At this stage I think it's too early to sort of justify the need of a NoSQL over SQL. Also do let me know if you would like to pitch in with helping with development of the API service.

[techdevsynergy]

@parthasarathydNU i dont have lot of free time since i work in very early startup but would like to contribute with respect to backend design. I saw for frontend you are going with nextjs what you are thinking for backend stack ? i think python based right ?

[parthasarathydNU]

@dharsan18 - Since there isn't much of Machine Learning / Data manipulation required for the API service, I am thinking of going ahead with either a Node / Java based backend. I wouldn't mind going ahead with a Python based web service as well, however I have better familiarity with Node. All tasks that require LLMs will be through API calls to Open AI's service so I don't see any particular reason to pick a stack apart form familiarity with the framework / language. Open to hear your thoughts. Further since this will be a separate service, we'll have to start a separate Repo for the API. I understand you are busy with work, but if you can pitch in - in terms of doing a QC on the PRs, design decisions related to the backend. That is much appreciated as well 😄

[parthasarathydNU]

Created a separate REPO for the AutoGrader API: https://github.com/autograder-org/autoGrader-api

Open to feedback on the ReadmeFile:

• Is it easy to set up the local API dev environment by following steps on the readme file
• Any feedback on the coding standards / folder structure followed in the API project