search

Samagra-Development / Doc-Generator

Create PDFs from a variety of formats.
17 stars 46 forks source link

[C4GT] Upgrade of Doc Gen backend to use Templater #121

Open karntrehan opened 1 year ago

karntrehan commented 1 year ago

Project Details

Doc Generator is a tool to create PDFs from a variety of formats. It is used extensively in our programs where we need to share a PDF report of some data. Be it class reports, student reports or usage numbers of our solutions.

Templater a REST API service that can store, manage and render templates. A Template is like a well-defined format in which data can be entered. The output generated is according to the format defined in the template. Templater helps us simplify and enhance this process by providing multiple services such as Data transformation, Lambda API, Templater Playground, Support for multiple engines, etc.

Currently doc generator has it's own backend. Moving to Templater as a backend will allow us to benefit from all the development happening at templater in real time. It would also allow us to spend time on features other than input processing which would be powered by templater.

Features to be implemented

Upgrade the backend to use templater removing custom logic handling at doc generator side.

What exists

A custom backend implementation of doc generator exists where it processes data, defines templates and generates PDF reports.

What needs to be built

  1. Create modules in templater.
  2. Leverage modules in 1 to orchestrate Doc Generator Workflows.
  3. Port existing functionalities from older codebase to NestJS.

Learning Path

Complexity

High

Skills Required

HTML, Python, Typescript, NestJS.

Name of Mentors:

@AnshulMalik

Project size

8 Weeks

Product Set Up

https://github.com/Samagra-Development/Doc-Generator#try-it-eyes

Acceptance Criteria

Out of Scope

  1. Additional functionality in Doc generator

Milestones

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition. C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

UdAyAn123 commented 1 year ago

I am looking to contribute please guide me.

Amisha0102 commented 1 year ago

Experienced in document generation, I'm eager to contribute to the project. Requesting guidance to ensure a smooth working process. Committed to delivering excellence, I'm open to learning from the team's expertise and achieving project goals together.

aditash20 commented 1 year ago

Hello sir I am Aditya Sharma a thrid year undergraduate at IIT kanpur. Sir i would like to work on this project, can you please tell me the further steps for the process.

radhay-1199 commented 1 year ago

The roadmap planned for this project is mentioned here.

aditash20 commented 1 year ago

@radhay-1199 I went through the roadmap. I have a question why are we migrating from Python to NestJS. Besides that I studied the roadmap and if their are any good first issues to work on, I would love to start working on them.

heyhimansh commented 1 year ago

Hey @karntrehan , I would like to contribute to this project . Looking forward to write a proposal for that ..

radhay-1199 commented 1 year ago

@aditash20 The decision to migrate from Python to NestJS for the doc generator service has been driven by several factors. NestJS is built with TypeScript, which is a statically typed superset of JavaScript. TypeScript offers advantages such as static type checking, enhanced code readability, and improved developer tooling. If the team is more comfortable or experienced with TypeScript and prefers its features, migrating to NestJS can align with their preferences.NestJS is built on top of Node.js, which is known for its scalability and non-blocking I/O model. If the doc generator service needs to handle high traffic or process large amounts of data, the asynchronous and event-driven nature of Node.js can provide performance benefits. Also, NestJS has a growing ecosystem and an active community, which means there are a variety of plugins, libraries, and resources available for developers. This can make development tasks more efficient and allow for easier integration with other services or frameworks. NestJS follows a modular and opinionated architectural pattern, inspired by Angular. It provides decorators, decorators-based routing, and dependency injection, which can help in organizing and maintaining code. If the existing codebase in Python was becoming difficult to manage or lacked a structured architecture, migrating to NestJS can address those concerns.

As for the good first issue, I guess you should try tickets based on implementing interfaces and types for the plugins and modules. This way you will be able to grasp quickly on core concepts and advantages of type-based programming.

FirePing32 commented 1 year ago

Hey team I am Prakhar Gurunani, a pre-final year student at BITS Pilani. I have worked on some Nest.JS projects and have ported some apps to TypeScript in the past. I am eager to contribute to this project. Looking forward to submitting a proposal soon 🚀

amberIS01 commented 1 year ago

Hey @karntrehan , I would like to contribute to this project .

FirePing32 commented 1 year ago

What needs to be built

  • Create modules in templater.
  • Leverage modules in 1 to orchestrate Doc Generator Workflows.
  • Port existing functionalities from older codebase to NestJS.

@AnshulMalik Can you explain this a bit more briefly ? Specifically the first 2 points.

Mounika-Pavanthi commented 1 year ago

I am excited to contribute to the migration project of the Doc Generator backend to Templater. With my experience in backend development and REST API services, I am well-equipped to support this transition and ensure a seamless integration. By leveraging Templater's features and services, we can enhance the functionality and efficiency of the Doc Generator tool. I am dedicated to collaborating with the team and delivering high-quality code to enable smooth data processing and PDF generation. Please consider my involvement in this project, as I am eager to contribute to its success and improve the overall user experience.

Bkmakwana2002 commented 1 year ago

Hello @karntrehan , @radhay-1199 , @FirePing32

I am an undergraduate student from IIT Ropar. I have gone through the roadmap and I have the required skills and substantial exposure to working with Javascript and its framework. Hope to contribute to this project. Kindly guide me in further procedures.

Regards Bhavya Kumar Makwana

radhay-1199 commented 1 year ago

Templator Enhancement Proposal: Package Transformation with Flexible Database Configuration

Objective

Transform our Templator microservice into a downloadable Node.js package, which can be utilized as a local library. The redesigned Templator will offer an optional database configuration at initialization, allowing the use of a user-provided relational database or a locally configured SQLite database for template management.

Current System

Currently, Templator is deployed as a microservice and relies on a PostgreSQL database for template storage. Templator's functionality includes creating, storing, and retrieving templates, and processing them with real-time data. Despite being effective, it can potentially introduce network latency when used frequently, and managing separate microservices can be complicated.

Proposed System

The new Templator will be a Node.js package available for download. After installation, users will initialize it in their applications, at which point they can provide a database configuration or leave it blank.

Here are the proposed changes:

Benefits

Conclusion

This proposal aims to make Templator more user-friendly and flexible. By providing both SQLite and PostgreSQL support, it caters to a wide range of use cases and allows for easier integration into various projects. By offering the syncDatabase function, users have control over when the database is set up, ensuring they can prepare adequately. We believe this transformation will make Templator more versatile and efficient, providing users with a better experience.

Tradeoffs

It's important to clearly document the behavior of your library in both scenarios (SQLite vs. PostgreSQL) so users know what to expect. For instance, SQLite doesn't support all the features and data types of PostgreSQL, so if users switch from one to the other, they might encounter issues.

FAQ

Why adding a separate trigger for syncDatabase and not just triggering it internally whenever class initalized?

For these reasons, it's better to provide a separate method to sync the database and to let the users call it when they see fit.

Instead, we could perform a simple check in the constructor to verify if the necessary tables exist, and if not, log a message to the console informing the user to run the syncDatabase() method. This way, we are not performing any potentially risky operations automatically, but we are still informing the user about the necessary actions to take.

Rough Implementation for the idea:(Thanks to GPT 🪨 )


const { PrismaClient } = require('@prisma/client');
const path = require('path');

class Templator {
  constructor(config) {
    if (!config.dbConfig) {
      // If no dbConfig is provided, default to SQLite
      this.prisma = new PrismaClient({
        datasources: {
          db: {
            url: 'file:./templator.db',
          },
        },
      });
    } else {
      // If dbConfig is provided, use that to connect to the PostgreSQL database
      this.prisma = new PrismaClient({
        datasources: {
          db: {
            url: config.dbConfig,
          },
        },
      });
    }

    // Attempt connection and table existence check
    this.prisma.$connect().then(() => {
      this.prisma.template.findMany({ take: 1 })
        .then(() => console.log('Database is in sync'))
        .catch(() => console.log('Database is not in sync. Please run templator.syncDatabase() to sync it.'));
    });
  }

async syncDatabase() {
    // Here, you would call the Prisma migrate script
    // However, this is tricky because Prisma Migrate is a CLI tool and does not currently provide a programmatic API.
    // A workaround would be to spawn a child process to run the CLI command:
    const { spawn } = require('child_process');
    const migrate = spawn('npx', ['prisma', 'migrate', 'dev', '--name', 'init']);

    migrate.stdout.on('data', (data) => console.log(`stdout: ${data}`));
    migrate.stderr.on('data', (data) => console.error(`stderr: ${data}`));
    migrate.on('close', (code) => console.log(`child process exited with code ${code}`));
  }
}