[RFC]: generate web documentation from JSDoc comments

USERSATOSHI commented 6 months ago

Full name

Tushar Bharti

University status

Yes

University name

Indian Institute of Information Technology, Bhagalpur

University program

Computer Science and Engineering

Expected graduation

31-5-2025

Short biography

Hello, My name is Tushar Bharti, A 3rd year undergrad student at IIIT Bhagalpur pursuing Computer Science and Engineering. I am a passionate developer with passion in software development , having proficiency in C/C++, JavaScript/TypeScript, NodeJS as well as experience in full stack development via academic and internship projects.

My skills span across various domain such as full stack development, application development, package/library development as well as deep understanding in various programming languages. I also compete regularly in competitive programming contests on Codeforces having rating of 1414 ( specialist ) which enhances my problem solving abilities and performant implementation of problems.

My general interests lies in testing out new frameworks as well as languages, such as trying out bunJS, rust, zig etc as well as frameworks like elysiaJS , svelte, astro etc. I also keep have huge interest towards parsers in compiler design, OOP and computer networks. Compiler design allowed me to write parsers for various data extraction tasks from code using code patterns , whereas I used OOP as a general way to implement structures and reusable instances and computer networks for implementing various protocols, such as udp , tcp etc

Timezone

IST ( UTC +5:30 )

Contact details

email:tushar.bharti.003@gmail.com,github:usersatoshi

Platform

Linux

Editor

I rely on two main code editors: Visual Studio Code (VSCode) and Neovim. Each one has its own perks, so I pick the one that fits the task.

When it comes to activities such as coding contests or conducting tool and integration tests, I find Visual Studio Code (VSCode) to be my preferred choice. It has a lot of extensions created by users for various integrations like language tools such as bunjs toolings as well a companion judge which i majorly use during competitive programming contests.

When going through larger projects or creating personal projects, I prefer to use Neovim as my primary editor. Its fast boot up time and fast navigation system due to fuzzy search makes going through files quick and less time consuming. Additionally, Neovim's has minimal battery consumption on my laptop, which is particularly great during battery use and traveling.

Having both editors allows me to switch between them as the situation arises.

Programming experience

In my programming journey, I've explored various domains, including competitive programming, software development, and web development.

I'm actively engaged in competitive programming, with a rating of 1414 (Specialist) on Codeforces, where I primarily use C++.

Aside competitive programming, I've delved into web development with Node.js and TypeScript. Some of my favorite projects include:

aoi.js, a Discord framework featuring its own scripting language, offering both interpretation and compilation capabilities. I worked on creating the core features such interpreter, compiler and a bundler as well as tooling system and plugins support for it.
aoi.db, a TypeScript-based database system I built from scratch, supporting widecolumnar and key-value databases to deepen my understanding of database architecture.

In addition to these projects, I've developed utilities tools like ytdn, a minimalist YouTube downloader CLI using Node.js.

Furthermore, I've gained practical experience through internships, where I contributed to a full-stack SaaS website with a web builder. Additionally, I've led academic projects, including creating official websites for my college fest and managing backend systems for my college's student academic portal.

JavaScript experience

In Javascript, I have gained experience via academic projects , Open source communities and writing applications. I have developed interactive web applications, backend using runtimes like nodejs and bunjs. I'm proficient in concepts like such as promises , async/await, closures, oops , es6 features, etc. This enabled me to write readable and user-friendly code with documents as a guide for others while trying to keep it as performant as possible. Furthermore, I have also experience using popular frameworks such as ReactJs, Astro, as well as runtimes such as NodeJs and BunJs.

My favorite features in Javascript are Promise.all and TypedArrays.

Promise.all allows to concurrently call the promises present in array as this can be used to concurrently save data into multiple files / fetching multiple apis allowing me to concurrently save data separated via hash into multiple files or a way to fetch multiple video streams in a playlist and concurrently save them in a folder while implementing playlist download option for the ytdn-cli while
- TypedArrays provides a great way to write and manage binary data which i found great while implementing column system for database.

My least favorite features in Javascript are Type Coercion and this.

Type Coercion in javascript had cause some weird bugs to happen while applying operations on them which made debugging for errors very tedious.
this keyword was also one of the least liked feature when i first started OOP in JavaScript as when used with es6 arrow functions it cause huge confusion on what the this is referenced to , though now I have enough knowledge on how this works and how it is manupulated, it is still one of my least liked

Node.js experience

In NodeJS, I have gained experiences by

creating backend systems in NodeJS & ExpressJs,
writing command-line interfaces via process.stdin.on then shifted to readline,
creating npm packages such as discord frameworks, database,etc ,
developed a visual studio code extension,
and desktop applications via electron.

C/Fortran experience

tho I primarily use C++ as my low level system language, I have solid foundation in C as this was the primary language that has been taught in our college along side with various projects ranging from implementing data structures and algorithms of C++ in C, to managements systems via command line , to embedded developmentn in 8051 micro-controller. thus giving me ample knowledge and understanding on it.

Interest in stdlib

stdlib (/ˈstændərd lɪb/ "standard lib") is a standard library for JavaScript and Node.js, with an emphasis on numerical and scientific computing applications. The library provides a collection of robust, high performance libraries for mathematics, statistics, data processing, streams, and more and includes many of the utilities you would expect from a standard library.

This encompasses 2 of my favorites: Mathematics and Javascript development. One of my favorite features is the wide range of implementations of mathematical functions available in Stdlib. These functions not only provide essential utilities for mathematical operations but also demonstrate the library's dedication to cover major mathematical / numerical computation programs as well as utilities functions

Version control

Yes

Contributions to stdlib

As of right now, I have 2 contributions in stdlib , one merged and one unmerged:

merged:

https://github.com/stdlib-js/stdlib/pull/1947

unmerged:

https://github.com/stdlib-js/stdlib/pull/2040

Goals

stdlib relies heavily on JSDoc comments to document its source code. Currently, the project has only rudimentary support for generating HTML docs from those comments. The goal of this idea would be to

Write an in-house JSDoc parser.
Generate HTML documentation from the parsed comments which is capable of supporting project conventions and its embrace of radical modularity.

JSDoc comments are oriented toward JavaScript source files; however, stdlib also uses similar documentation practices for documenting C source files and make files. A possible extension to the in-house JSDoc parser would be to support these other source file types. As those file types may require separate AST parsers, supporting other file types is likely to require writing separate comment parsers for each source type.

Why this project?

generate web documentation from JSDoc comments requires in-depth knowledge in creating a lexical parser as well as an syntax parser which then would be used to generate HTML + CSS files for web documentation. these parser would require breaking down jsDoc comments into useful tokens which then would be converted into an Abstract Syntax Tree ( AST ).

As I have a background in Computer Science and coursework in Compiler Design and experience in web development, this project excites me to create a working parser which constitute the frontend part of compiler design and then use the generated AST to create static website documentation which would test both my knowledge in compiler design as well as experience in web development.

Qualifications

i have solid experience in Javascript, NodeJS, C/C++ as well as foundations required for software development as well as adaptability to learn and understand codebase quickly with quick initial helps as guides. This allows me to understand and write programs efficiently.

As a 3rd year undergraduate at IIIT Bhagalpur pursuing computer science, I have coursework in compiler design as well as various academic and internship projects in web development which makes me well suited for this project.

Being my favorite subject , Compiler Design allowed me to write different parsers ranging from

extracting metadata from function such as params , return type , required and optional fields based on coding style patterns,
modifying typedoc's doc site generation script by injecting my own script ,
to creating a custom scripting language which is both interpreter-able as well as compile-able to JavaScript and so on.

Plus handling web projects for my college such as Student Academic Portal , Techno-cultural fest's official site, etc as well as internships allowed me having deep knowledge in web development.

Also having experience through competitive programming via Codeforces as well as collaborative projects allows me to approach this project in a way which tries to improve the efficiency of the parser but also keeps the code human-readable to allow for easier and manageable maintenance and up-gradation.

Prior art

Here are the implementation of a parser for JsDocs and its relatives which I found during research:

https://github.com/DavidWells/jsdoc-parser ( updated fork of dox )
https://typedoc.org/ ( documentation generator for typescript which supports tsDoc tags )
https://www.doxygen.nl/index.html ( documentation generator for C/Cpp files )
https://github.com/jsdoctypeparser/jsdoctypeparser ( parser to parse typedefs )
https://github.com/tj/dox ( documentation generator for Javascript using jsDoc tags )
https://github.com/eslint/doctrine ( Eslint old jsDoc parser )
https://github.com/gajus/eslint-plugin-jsdoc ( community version of doctrine recommended by eslint )

these resources should be enough to implement an in house jsDoc parser as well as document generation from that parser

Commitment

1 May - 26 May -> Bonding Period 27 May - 7 July -> 35 hours/week ( 35 6 ) 8 July - 17 August -> 23 hours/week ( 23 6 ) Total = 210 + 138 = 348 hours

Schedule

Assuming a 12 week schedule,

Community Bonding Period: During this period, I will work to explore the project's codebase. Checking out various guidelines and practices mentioned for contributors, trying to familiarize with the project's jsDoc implementation as well as usage pattern. I will also check out the jsDoc utilities listing out the custom types , implementation and transformations

Week 1 & Week 2: Base Layout:

note down all custom implemented types and scan the coding style pattern in which jsDoc is being used, so that a set of regular expression can be derived to split the comments into useful tokens.

implement a typedef parser which can parse existing type definitions as well as Generic and custom types into nodes.

the possible outcome for this typedef parser would give

// #Usage
typeDefParser(type: string): IType

// #Example
typeDefParser("Array<string>")
/* returns {
original: "Array<string>",
   parsed: "Array<#Child_0#>",
   type: "Generic",
   name: "Array",
   isCustom: false,
   link: "link to mdn array",
   body: [{
       original: "string",
       parsed: "string",
       type: "primitive",
       name: "string",
       isCustom: false,
       body: [],
       link: "link to mdn string"
   }]
}
*/

typeDefParser("number")
/* returns {
   original: "number",
   parsed: "number",
   type: "primitive",
   name: "number",
   isCustom: false,
   body: [],
   link: "link to mdn number"
}
*/

Week 3 & Week 4:

Lexical parser:

create a lexical parser to split jsDocs into literals as well as a function parser to extra additional metadata from functions.
create a symbol table that tracks references such as function name, file path , line number etc which would be used for referencing source code with documentation

Syntax Parser:

discuss with mentors to define the needed data as well as additional data for the parser to store in abstract syntax tree

create a syntax parser to convert the lexical tokens into an abstract syntax tree

Example for the Syntax parser would be:

// using @stdlib/math/base/special/fast/abs as example

// # Usage
generateJavascriptAST(jsFilePath: string): JSAST

// # Example
generateJavascriptAST("./lib/node_modules/@stdlib/math/base/special/fast/abs/lib/index.js")
/* returns {
    metadata: {
        package: "@stdlib/math/base/special/fast/abs",
    },
    data: [{
        functionName: "abs",
        startLine: "function abs( x )",
        tags: [{
            name: "Description",
            content: "Computes the absolute value of `x`."
        },{
            name: "params",
            type: {
                original: "number",
                parsed: "number",
                type: "primitive",
                name: "number",
                isCustom: false,
                body: [],
                link: "link to mdn number"
            },
            value: "x",
            content: "input value"
        },{
            name: "returns",
            type: {
                original: "number",
                parsed: "number",
                type: "primitive",
                name: "number",
                isCustom: false,
                body: [],
                link: "link to mdn number"
            },
            content: "absolute value"
        },{
            name: "example",
            content: "var v = abs( 2.0 );\n// returns 2.0"
        },{
            name: "example",
            content: "var v = abs( 0.0 );\n// returns 0.0"
        },{
            name: "example",
            content: "var v = abs( -0.0 );\n// returns -0.0"
        },{
            name: "example",
            content: "var v = abs( NaN );\n// returns NaN"
        }]
    }]
}
*/

Week 5 & Week 6: (midterm)

HTML Generation:
- discuss the html layout and format with the mentor to get a basic idea on how to site would look like.
- writing html scripts to generate website from the given ast from syntax parser.
- using symbol table to convert all possible reference-able paths into links.
- writing css to style the documents such as syntax highlight , responsive design etc.

Week 7 & Week 8:

MakeFile Parser Extension:

extend the implemented parser for MakeFiles

Update Html Script to accommodate for new add-ons as well linkers to link dependencies pages with each other

Example for the Syntax parser would be:

// using @stdlib/math/base/special/fast/abs as example

// # Usage
generateMakefileAST(filePath: string): MAKEFILESAST

// # Example
generateMakefileAST("./lib/node_modules/@stdlib/math/base/special/fast/abs/src/Makefile")
/* returns {
    metadata: {
        package: "@stdlib/math/base/special/fast/abs",
    },
    data: [{
        metadata: {
            file: "./lib/node_modules/@stdlib/math/base/special/fast/abs/src/Makefile",
            LNO: 57,
            dependencies: [],
        },
        functionName: "clean-addon",
        startLine: "clean-addon",
        tags: [{
            name: "Description",
            content: " Removes generated files for building an add-on."
        },{
            name: "example",
            content: "make clean-addon"
        }]
    },{
        metadata: {
            file: "./lib/node_modules/@stdlib/math/base/special/fast/abs/src/Makefile",
            LNO: 68,
            dependencies: [],
        },
        functionName: "clean",
        startLine: "clean",
        tags: [{
            name: "Description",
            content: " Removes generated files."
        },{
            name: "example",
            content: "make clean"
        }]
    }]
}
*/

Week 9:

C parser module:

writing a C parser for generating ast from C comments format

converting the system into a bridge/adapter pattern

Example for the parser would be:

// using @stdlib/math/base/special/fast/abs as example

// # Usage
generateCAST(filePath: string ): CAST

// # Example
generateCAST("./lib/node_modules/@stdlib/math/base/special/fast/abs/src/main.c")
/* returns {
    metadata: {
        include: "stdlib/math/base/special/fast/abs.h",
        LNO: 31,
        dependecies: [],
        constants: [],
        licenseYear: 2024
    },
    data: {
        functionName: "stdlib_base_fast_abs",
        startLine: "double stdlib_base_fast_ampbm( const double x, const double y )",
        tags: [{
            name: "param",
            value: "x",
            type: "const double",
            content :"input value"
        },{
            name: "return",
            type: "double",
            content: "absolute value"
        },{
            name: "example",
            content: "double v = stdlib_base_fast_abs( -1.0 );\n// returns 1.0",
        }]
    }
} 
*/

Week 10:

C HTML Generation:
- generate html documentation for C Apis
- create/ update make recipe to add this
Week 11 & Week 12:

Testing, Documentation & Extension :
- this week will be mainly for fixing bugs and edges cases where the parser could fail
- optimize the generation speed and time
- check if it is possible to extend this parser to different repositories and other languages
- Write documentation on how parser works , internal functions as well as examples
Final Week:

I will try to complete and wrap up all the things during this week and submit it. Notes:
The community bonding period is a 3 week period built into GSoC to help you get to know the project community and participate in project discus familiarize and setup sion. This is an opportunity for you to setup your local development environment, learn how the project's source control works, refine your project plan, read any necessary documentation, and otherwise prepare to execute on your project project proposal.
Usually, even week 1 deliverables include some code.
By week 6, you need enough done at this point for your mentor to evaluate your progress and pass you. Usually, you want to be a bit more than halfway done.
By week 11, you may want to "code freeze" and focus on completing any tests and/or documentation.
During the final week, you'll be submitting your project.

Checklist

[X] I have read and understood the Code of Conduct.
[X] I have read and understood the application materials found in this repository.
[X] I understand that plagiarism will not be tolerated, and I have authored this application in my own words.
[X] I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
[X] The issue name begins with [RFC]: and succinctly describes your proposal.
[X] I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

Pranavchiku commented 6 months ago

Hey @USERSATOSHI, this proposal is good. Impressed by your skills, I have few suggestions

You may brief about what will be the grammar for your parser, provide an example, develop a small parser that for example works for a particular package ( say C example documentation ), describe it in your proposal.
Currently, the proposal lacks "how" & "what" are you going to do. Adding this will make it great.

USERSATOSHI commented 6 months ago

Hey @USERSATOSHI, this proposal is good. Impressed by your skills, I have few suggestions

* You may brief about what will be the grammar for your parser, provide an example, develop a small parser that for example works for a particular package ( say C example documentation ), describe it in your proposal.

* Currently, the proposal lacks "how" & "what" are you going to do. Adding this will make it great.

Hi @Pranavchiku , So should i mention an actual snippet or an example for each

like for typedef parser


TypeDefParse("Array<string>")
/* returns { 
  original: "Array<string>", 
  parsed: "Array<#Child_0#>",
  type: "Generic",
  name: "Array",
  isCustom: false,
  link: "link to mdn array",
  body: [{
    original: "string",
    parsed: "string",
    type: "primitive",
    name: "string",
    isCustom: false,
    body: [],
    link: "link to mdn string"
 }]
}
*/

TypeDefParse("(string | number)[]")
/* returns {
   original: "(string | number)[]",
   parsed: "(#CHILD_0# | #CHILD_1#)[]",
   type: "Generic",
   name: "Array",
   isCustom: false,
   link: "link to mdn array",
   body: [{
       original: "string",
       parsed: "string",
       type: "primitive",
       name: "string",
       isCustom: false,
       body: [],
       link: "link to mdn string",
   },{
       original: "number",
       parsed: "number",
       type: "primitive",
       name: "number",
       isCustom: false,
       body: [],
       link: "link to mdn number",
   }]
}
*/

Pranavchiku commented 6 months ago

Yes, take a package, mention every small requirement in detail.

USERSATOSHI commented 6 months ago

ok got it!

USERSATOSHI commented 6 months ago

@Pranavchiku I have updated the the schedule can you review it back in your free time

kgryte commented 6 months ago

Thanks, @USERSATOSHI, for sharing a draft of your proposal. A few comments:

The parser should not be extended beyond JSDoc and JSDoc-like comments. So, e.g., the project should not, e.g., resolve the list of module dependencies or do anything with file "sections" (i.e., // MAIN //, etc).
We heavily use JSDoc comments in three source file types: 1) JavaScript, 2) C, and 3) Makefile. You address the first two. I suggest including the third and swapping in for the "extensions" you mention in weeks 7&8.
Another reference implementation it may be worth investigating is how ESLint and its associated plugins handle JSDoc parsing. And IIRC, another parser is doctrine, but this has been archived: https://github.com/eslint/doctrine. You may want to find out what is used more recently.

USERSATOSHI commented 6 months ago

@kgryte Hi! Thank you for reviewing, about the extension , we can cut off most of the thing but dependencies can be left in so that we can make a reference link on require like you suggested on the original idea issue:

Generated JSDoc documentation is likely to be static, but one can envision possible extension to supporting inter-package linking (e.g., require statements would be transformed into links). Link

We heavily use JSDoc comments in three source file types: 1) JavaScript, 2) C, and 3) Makefile. You address the first two. I suggest including the third and swapping in for the "extensions" you mention in weeks 7&8.

I will update the week 7 & 8 to include MakeFile

nother reference implementation it may be worth investigating is how ESLint and its associated plugins handle JSDoc parsing. And IIRC, another parser is doctrine, but this has been archived: https://github.com/eslint/doctrine. You may want to find out what is used more recently.

Thank you for the reference, I will check out eslint and doctrine as you mentioned and do some investigation from here too

stdlib-js / google-summer-of-code