[RFC]: Achieve feature parity with async.js

[nightknighto]

Full name

Ahmed Atwa

University status

Yes

University name

Ain Shams University

University program

Computer Engineering

Expected graduation

2025

Short biography

I am Ahmed Atwa, a Computer Engineering student from Egypt, and a full-stack JavaScript developer with focus on MERN stack with TypeScript. I've been in the open-source scene for quite some time, having been in MLH Fellowship program for 3 months, contributed to multiple open-source organizations, made a few open-source projects and more.

I'm highly skilled in Fullstack web development, primarily with React in the frontend and Nodejs in the backend. My skills include:

• TypeScript, ReactJS, Redux, Tailwind, Bootstrap, ElectronJS, a little bit of Angular.
• NodeJS, Express, MongoDB, a little bit of NestJS.
• AWS, Github Actions, CI/CD.
• Software Engineering designing, unit and integration testing, and more.
• Some knowledge of python, C & C++.

For my technical experience:

• MLH Fellowship Program as mentioned, where I worked with the company OpenSauced on their projects for 3 months.
• After the program, I was hired and continued working part-time with OpenSauced for 3 more months.
• I also am very involved in campus clubs, and have been the leader of fullstack development in one club, mentoring and leading my team to successfully creating the web projects the club needed.
• I've also been working part-time with a private contractor on his Angular-NodeJS application for 4 months.

Further more, aside from working in companies, I've been making my own projects for the past 3 years, finding my passion in the field through a huge variety of different personal projects that all use web technologies, ranging from a desktop application that designs user interface through drag-and-drop, to game development systems made with TypeScript, to an educational resource gatherer kind of website for my colleagues, and many many more that you can find in my Github.

Timezone

GMT +2

Contact details

https://github.com/Deadreyo, ahmedatwa866@yahoo.com, www.linkedin.com/in/ahmed-mohamed-atwa, NightKnight on Element

Platform

Windows

Editor

VSCode, the typical and best editor for web development.

Programming experience

For my technical experience:

• MLH Fellowship Program where I worked with the company OpenSauced on their projects for 3 months.
• After the program, I was hired and continued working part-time as Fullstack developer with OpenSauced for 3 more months.
• I also am very involved in campus clubs, and have been the leader of fullstack development in one club, mentoring and leading my team to successfully creating the web projects the club needed.
• I've also been working part-time with a private contractor on his Angular-NodeJS application.

Furthermore, aside from working in companies, I've been making my own projects for the past 3 years, finding my passion in the field through a huge variety of different personal projects that all use web technologies, including:

• A desktop application that designs user interface through drag-and-drop and transforms it into code.
• Game development systems made with TypeScript
• An educational resource gatherer kind of website for my colleagues
• A backend for OpenSauced that generated social images for their users on demand.
• A full-fledged fullstack Recruitment System for a campus club, consisting of a Landing page, Dashboard and the Backend.
• A desktop app for quick utilities on folders with NodeJS. and many many more that you can find in my Github.

JavaScript experience

I am a "JS-centric" developer, meaning that everything I do is in JavaScript or TypeScript. Whatever it is I want to do I simply search for a way to do it in JS, which includes:

• Desktop applications I built using JavaScript. Ex: https://github.com/Deadreyo/Warcraft-3-Reforged-UI-Designer
• Automation scripts I built using NodeJS. Ex: https://github.com/MLH-Fellowship/prep-project-22.AUG.PREP.2/tree/main/scripts
• Of course the backend and the frontend applications as well. Ex: https://github.com/Deadreyo/ASUENG-Library , https://github.com/Deadreyo/SemiColon-dashboard-frontend
• Much more uncommon for JS, I used to make game mods and systems using TypeScript that were converted into LUA. Ex: https://github.com/Deadreyo/WC3-UI-Minigames

So due to all that, I'd say I'm completely comfortable with JavaScript and know a lot about it with a lot of hands-on experience.

My favourite thing about it is that it is extremely flexible, allowing you to experiment and try any design pattern you want, any programming paradigm, basically no restrictions in how you write your code and design stuff. Add TypeScript for type-checking and you get the perfect dream language in my opinion.

My least favourite thing about JavaScript is that it hasn't invaded the whole world yet.

Node.js experience

A lot of experience in NodeJS, whether through:

• using it in the backend applications along with Express or NestJS.
• using it to build Desktop Applications with ElectronJS.
• creating automation scripts with it to be run by CI.

Some of the most notable experiences with it:

• Creating a backend application for OpenSauced company that generates social images on demand. This used NodeJS, NestJS and many libraries to make it possible. https://github.com/open-sauced/opengraph
• Created a desktop application called "Folder Utilities App", that basically can do multiple actions on a folder of your choosing, such as compressing all images inside that folder, or extracting all deeply-nested files and such. This was made with NodeJS and a lot of file-system manipulations of course. https://github.com/Deadreyo/Folder-Services-Utils-App
• Created a desktop application that can analyze a directory with a special structure and generate a JSON file for it, mainly used to convert a group of educational files into JSON to be uploaded and shared on a website. https://github.com/Deadreyo/ASUENG-Manager
• Of course in addition to all the backend applications I've worked on. https://github.com/Deadreyo/SemiColon-backend
• Most of the projects linked in the JavaScript section also applies here, as they all use NodeJS.

C/Fortran experience

I know limited C, worked on some Embedded Systems projects from college. I also know C++, used to be very passionate about Arduino and electronics and making projects with arduino with C++.

Interest in stdlib

What caught my interest in stdlib that was different from other organizations, is how extremely well everything is documented and structured. There is a huge noticeable emphasis on documentation, whether through READMEs, the examples or the JSDocs, the whole ecosystem is really inviting to dive into and contribute to it and I loved that so much.

The idea of stdlib itself is also something that can have a huge effect on so many developers, and so contributing to such a project means that your effect can extend to the lives and work of thousands of developers. I don't like anything more than to work on something that can have a big impact on so many people.

Version control

Yes. Mainly use Git in every project.

Contributions to stdlib

Currently I have 3 open PRs, all of which are awaiting review.

• My latest and main PR is a package that is part of the project I am writing the application for: utils/async/parallel. It implements a new package that runs async functions in parallel. All details are listed and were discussed in the issue linked inside the PR. https://github.com/stdlib-js/stdlib/pull/1896

• A previous PR was: update blas/ext/base/dapxsum to follow current project conventions https://github.com/stdlib-js/stdlib/pull/1793

• My first PR was: improve README examples of stats/base/dists/normal namespace https://github.com/stdlib-js/stdlib/pull/1679

Goals

Overview

The goal of this project will be to have a complete set of async APIs for all kind of various operations. By reaching a 1:1 feature parity with async.js APIs, stdlib will be a perfect replacement for async.js and the need for installing additional libraries alongside stdlib will be decreased further and further.

Async APIs are a very important part of the JavaScript ecosystem, and having a complete set of them in stdlib will be a huge step forward for the project and the community. Not only are most of the APIs in async.js very useful, but also the fact that they are async means that they are very important for performance and for the modern web.

Deliverables

Refactoring packages to enable Observability

Currently there are 36 packages inside utils/async that can potentially be refactored to return status objects that can indicate the status of each async function execution or each step.

Whether this will be pursued or not, and whether all packages would be refactored or a set of them, is to be decided by the maintainers.

Directly inspired by node-vasync, the status object returned from the functions or passed to the callback could look like this:

status: { operations:
   [], // array storing the status of each async step, depends on what type of function. For `parallel` would be each function status.
  successes: [], // successful outputs
  ndone: 0, // number of successes
  nerrors: 0 // number of fails / errors
  }

Exact format is to be decided with the maintainers in the Community Bonding period, and may vary greatly depending on what package it is.

Adding new packages

Comparing the APIs of async.js with the APIs of stdlib, we can see that there are a lot of APIs that are missing from stdlib. The goal of this project will be to implement all of these APIs, and to make sure that they are all well-documented and tested.

Categorizing the APIs as in async.js, the APIs that are missing from stdlib are:

Collections

• concat: Applies a function to each item in collection, concatenating the results.
• detect: Returns the first value in a collection that passes an async truth test.
• each (including eachOf): Applies a function to each item in collection in parallel.
• filter: Returns a new array of all the values in collection which pass an async truth test.
• reject: The opposite of filter. Removes values that pass an async truth test.
• sortBy: Sorts a list by the results of running each collection value through an async function. Total: 7

Control Flow

• applyEach: Applies the provided arguments to each function in the array, calling callback after all functions have completed.
• auto: Determines the best order for running the async functions in tasks, based on their requirements.
• autoInject: A dependency-injected version of the auto function.
• cargo: Creates a cargo object with the specified payload. Tasks added to the cargo will be processed altogether
• forever: Calls an asynchronous function with a callback parameter that allows it to call itself again, in series, indefinitely.
• queue: Creates a queue object with the specified concurrency. Functions added to the queue are processed in parallel
• priorityQueue: The same as queue only tasks are assigned a priority and completed in ascending priority order.
• race: Runs an array of functions in parallel. Once any of the functions complete or pass an error to its callback, the main callback is immediately called.
• retry: Attempts to get a successful response from the function for x times before returning an error.
• retryable: This method wraps a function and makes it retryable, rather than immediately calling it with retries.
• tryEach: It runs each task in series but stops whenever any of the functions were successful.
• until: Repeatedly call a function until a condition returns true
• whilst: Repeatedly call a function, while a condition returns true. Total: 12

Utils

• apply: Creates a continuation function with some arguments already applied.
• asyncify: Take a sync function and make it async. This is useful for plugging sync functions async functions.
• constant: Returns a function that when called, calls-back with the values provided.
• dir: Logs the result of an async function to the console using console.dir to display the properties of the resulting object.
• ensureAsync: Wrap an async function and ensure it calls its callback on a later tick of the event loop.
• log: Logs the result of an async function to the console.
• memoize: Caches the results of an async function.
• nextTick: Calls callback on a later loop around the event loop.
• reflect: Wraps the async function in another function that always completes with a result object, even when it errors.
• reflectAll: A helper function that wraps an array or an object of functions with reflect.
• setImmediate: Calls callback on a later loop around the event loop.
• timeout: Sets a time limit on an asynchronous function.
• unmemoize: Undoes a memoized function Total: 13

This makes a total of 32 APIs that are missing from stdlib. The mentors will set the priorities to ensure that the most important APIs are implemented first, then the less important ones.

Note: Each of the API will include the variations from limit and series in the same API, as opposed to async.js where they are separate APIs.

Each of the APIs will be planned well and discussed with the mentors before implementation. Some may be redundant or not needed, as will be decided with the mentors.

The project can also include improving the existing APIs in stdlib to extend their functionality or add options to them.

Improving the existing APIs (if time allows)

• inmapAsync: Add the option to output in a new array instead of modifying the input array.
• parallel: Allowing the input to be an object instead of an array.
• Open to more suggestions from the mentors.

Why this project?

This project is a huge opportunity for me to practice JS in creating library APIs, and to learn a lot about the language and the ecosystem. It expands deeply into NodeJS, JS OOP, async programming, and the JavaScript ecosystem in general. Learning and working on APIs that are to be used by developers is the best way to master a language, and I am very excited to work on this project.

Qualifications

My qualifications stem from my experience in JavaScript and NodeJS that are very relevant to the project and are described in detail in the previous sections. This experience enables me to understand the project and its requirements, and to work on it effectively and efficiently. I also know very well how to create maintainable and clean code.

I have used JS async programming in many of my projects, and I have a good understanding of the language and its ecosystem. I have also worked on many open-source projects and have a good understanding of the open-source community and its standards.

I am also familiar with the stdlib project and its structure, and I have already contributed to it. This makes me a good fit for the project and enables me to work on it effectively.

Prior art

• This project and the APIs to be implemented are directly taken from async.js.
• I have already created the parallel API, which is the first API to be implemented in the project. This gave me a good understanding of the project and its requirements, and enabled me to scope them accurately.

Commitment

I will be having my final exams in the first 4 weeks of the coding period, so at that time I will be able to work for 1-2 hours daily. After that, I will be fully available for the project for the rest of the Summer and will be able to work full-time on it, ~40 hours/week, as much as needed to keep up with the project's timeline. The first 4 weeks will be compensated for by extending the project's timeline by 4 weeks, as allowed by the GSoC rules.

Schedule

Given uncertainty in how many of the missing APIs are desired to be added, whether the maintainers will decide to pursue the Observability refactoring and whether for all of the existing APIs or a set of them, it became much harder to plan a definite timeline for the deliverables given that many of the details are to be decided with the maintainers.

So, I will leave the old schedule (one which assumed only adding new APIs), and create a new schedule that will assume we want to refactor all 36 existing APIs for observability, and the rest of the program adding new APIs.

Old Schedule (No observability refactoring and implementing all new APIs)

Assuming a 16 week schedule (including the extended 4 weeks),

• Community Bonding Period:

In this period, I will be focusing on discussing all the deliverables with the mentors, planning each API in detail and opening an issue for each one that describes the requirements and the implementation notes for them. Similar to how I did with the parallel API.

In this period we will also decide on how exactly the timeline will go like, which new APIs are needed, which APIs (or all of them) will need observability, and more.

• Week 1 to Week 4: (Exams period)

In this period, I will continue to define any APIs left from the Community Bonding Period. I will also work on the less complex APIs that require less time to implement from the Collection category. This can also be a great period for continued discussion and prototyping for future work, for example converting a few packages and adding a few new APIs and investigate the implementation details and the understanding with the mentors, so that when the heavy schedule begins, everything is much clearer and will be more productive with less need for requesting changes.

• Week 5 and Week 6:

Implement all the APIs in the Collection category, fully documented, tested and reviewed.

• Week 7 to Week 11: (midterm on Week 10)

Implement all the APIs in the Control Flow category, fully documented, tested and reviewed.

• Week 12 to Week 14 or 15:

Implement all the APIs in the Utils category, fully documented, tested and reviewed.

• Week 15:

Implement any remaining APIs. This week will be used to catch up on any APIs that were not implemented in the previous weeks.

Work on improving the existing APIs.

• Final 2 weeks:

Finish off any remaining work, ensure all the APIs are fully documented, tested and reviewed, and that all PRs are merged. Open new issues for any bugs that were found during the project, or any improvements that can be made to the APIs. Submit the final report and the final evaluation..

New schedule (including Observability refactoring and limiting of new APIs)

Assumptions: 1- All 36 existing APIs are desired to be observable 2- The priority of observability on existing APIs is higher than the priority of creating new APIs.

Assuming a 16 week schedule (including the extended 4 weeks),

• Community Bonding Period:

In this period, I will be focusing on discussing all the deliverables with the mentors, planning each API in detail and opening an issue for each one that describes the requirements and the implementation notes for them. Similar to how I did with the parallel API.

In this period we will also decide on how exactly the timeline will go like, which new APIs are needed, which APIs (or all of them) will need observability, and more.

• Week 1 to Week 4: (Exams period)

Convert a few packages for observability. This can also be a great period for continued discussion and prototyping for future work, for example converting a few packages and adding a few new APIs and investigate the implementation details and the understanding with the mentors, so that when the heavy schedule begins, everything is much clearer and will be more productive with less need for requesting changes.

• Week 5 and Week 6:

Convert 9-12 packages for observability

• Week 7 and Week 8:

Convert 9-12 packages for observability

• Week 9 and Week 10: (midterm on Week 10)

Convert 9-12 packages for observability

• Week 11: (midterm on Week 10)

Fully finish the conversion for observability, ensure all observability is finished and merged. Week 11 can also include implementing new APIs.

• Week 12 to Week 14:

Implement the decided-on new APIs. Estimated 6-9 APIs depending on the complexity of each.

• Week 15:

Implement additional 3 new APIs.

• Final 2 weeks:

Finish off any remaining work, ensure all the APIs are fully documented, tested and reviewed, and that all PRs are merged. Open new issues for any bugs that were found during the project, or any improvements that can be made to the APIs. Submit the final report and the final evaluation..

My own notes:

• Each API will take on average 2-3 days to implement, document and test. This is a rough estimate and the actual time may vary depending on the complexity of the API and the requirements of the mentors.
• Refactoring each API for Observability may take around 1 day, given the need for changing the implementation, docs, tests, benchmarks, etc. and also depending on whether the status object format will be static or dynamic (i.e. whether can be done once and used as a reference)
• Schedule is leaning more to the worst case regarding how much time each API/refactor will take. The actual case can be much quicker and much more productive.
• Merging PRs and responding to requested changes are implicit in the schedule, as building out well-detailed issues to base development on should help greatly in reducing the number of changes needed.
• By the end of each week, a report of what was accomplished in that week will be given to mentors.

Notes:

• The community bonding period is a 3 week period built into GSoC to help you get to know the project community and participate in project discussion. This is an opportunity for you to setup your local development environment, learn how the project's source control works, refine your project plan, read any necessary documentation, and otherwise prepare to execute on your project project proposal.
• Usually, even week 1 deliverables include some code.
• By week 6, you need enough done at this point for your mentor to evaluate your progress and pass you. Usually, you want to be a bit more than halfway done. => week 10
• By week 11, you may want to "code freeze" and focus on completing any tests and/or documentation. => week 15
• During the final week, you'll be submitting your project.

Related issues

No response

Checklist

• [X] I have read and understood the Code of Conduct.
• [X] I have read and understood the application materials found in this repository.
• [X] I understand that plagiarism will not be tolerated, and I have authored this application in my own words.
• [X] I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
• [X] The issue name begins with [RFC]: and succinctly describes your proposal.
• [X] I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

[kgryte]

@Deadreyo Thank you for sharing your draft proposal. A few comments:

1. Extending the timeline to accommodate your exams should be fine.
2. As mentioned in the original idea issue thread, there is the idea of refactoring some of the existing APIs. In particular, this is motivated by https://github.com/TritonDataCenter/node-vasync and the idea of enabling observability. One of the issues with async.js is that one does not have an easy mechanism for knowing the current status or being able to query on an ad-hoc basic. The idea behind vasync is to share a stateful object that the library updates as results come in. We'd want to decide whether to pursue this refactoring, and if so, would likely want to set about updating the existing utils/async packages. As some of the APIs in your list we will not likely want to pursue, I believe we should be able to accommodate the refactoring in your proposed schedule.

[steff456]

@Deadreyo thanks for opening this draft proposal!

I have a few comments to add,

1. You should add some time in your schedule for reviews, specially in the last couple of weeks to make sure you are able to merge successfully all the PRs before the program ends not just one week.
2. I also feel that the amount of packages is ambitious, but as @kgryte already mentioned there may be some that will not be included so we can refine that later.

[nightknighto]

A lot of details are ambiguous like how many new APIs are needed, how many existing APIs need to be observable (if at all, as the idea still seems to be in consideration). This made it too hard to predict exactly on what I will be working on during the program and creating a schedule, so I've decided to keep the old one and create a new schedule with some assumptions. Hope that is okay, as I couldn't think of another solution.

2. As mentioned in the original idea issue thread, there is the idea of refactoring some of the existing APIs.

Included this in the new schedule.

1. You should add some time in your schedule for reviews, specially in the last couple of weeks to make sure you are able to merge successfully all the PRs before the program ends not just one week.

Updated.

Thank you both for your inputs!