Refactor Backend Project Structure for Improved Modularity and Maintainability

[1varunvc]

Description

Our backend codebase has grown, and the current flat file structure makes it challenging to manage and scale the application efficiently. To enhance maintainability, readability, and scalability, we need to reorganize the backend into a more modular structure. This refactoring will separate concerns, making it easier to add new features, fix bugs, and collaborate with other developers.

---

Tasks

1. Create a New Modular Project Structure

   Organize the backend directory into the following subdirectories:

   backend/
   ├── .env.example
   ├── package.json
   ├── server.js
   ├── config/
   │   └── config.js
   ├── routes/
   │   ├── index.js
   │   ├── spotifyRoutes.js
   │   └── authRoutes.js
   ├── controllers/
   │   ├── spotifyController.js
   │   └── authController.js
   ├── services/
   │   ├── spotifyService.js
   │   ├── spotifyClientCredentials.js
   │   └── authService.js
   ├── middleware/
   │   ├── rateLimiter.js
   │   ├── authMiddleware.js
   │   └── errorHandler.js
   ├── models/
   │   └── user.js
   └── utils/
      └── logger.js

2. Move Existing Files to Appropriate Directories

   • Environment Variables:
     • Update .env.example with all necessary environment variables.
   • Server Configuration:
     • Move existing server.js to the root of the backend/ directory.
   • Routes:
     • Move spotifyRoutes.js to routes/spotifyRoutes.js.
     • Move OAuth routes from auth.js to routes/authRoutes.js.
     • Create routes/index.js to aggregate all routes.
   • Controllers:
     • Create controllers/spotifyController.js and move route handlers from spotifyRoutes.js.
     • Create controllers/authController.js and move route handlers from auth.js.
   • Services:
     • Move spotifyClientCredentials.js to services/spotifyClientCredentials.js.
     • Move OAuth logic from auth.js to services/authService.js.
     • Create services/spotifyService.js for Spotify API interactions.
   • Middleware:
     • Create middleware/rateLimiter.js for rate-limiting middleware.
     • Create middleware/authMiddleware.js for authentication middleware.
     • Create middleware/errorHandler.js for global error handling.
   • Utilities:
     • Create utils/logger.js for logging utilities.
   • Models:
     • Create models/user.js for user schema (optional, for future use).

3. Adjust Code to Match the New Structure

   • Update Import Paths:
     • Adjust all require or import statements to reflect new file locations.
   • Module Exports:
     • Ensure that all modules are correctly exported and imported.
   • Server Setup:
     • Update server.js to use the new routes and middleware.
     • Initialize Passport.js in server.js and require necessary services.

4. Centralize Configuration

   • Create config/config.js:
     • Centralize access to environment variables and configuration settings.
     • Update code to use configuration from this file.

5. Implement Middleware Functions

   • Rate Limiting Middleware:
     • Move rate limiting logic to middleware/rateLimiter.js.
   • Authentication Middleware:
     • Move ensureAuthenticated function to middleware/authMiddleware.js.
   • Error Handling Middleware:
     • Implement global error handling in middleware/errorHandler.js.
     • Apply error handling middleware in server.js.

6. Isolate OAuth Code

   • Services and Controllers:
     • Keep OAuth-related code in services/authService.js and controllers/authController.js.
   • Routes:
     • Ensure OAuth routes are in routes/authRoutes.js and do not interfere with current functionality.
   • Future Use:
     • Document that OAuth code is retained for future implementation.

7. Update Environment Variables

   • Ensure Security:
     • Update .env.example with placeholders for all required environment variables.
     • Confirm that .env files are listed in .gitignore to prevent sensitive data from being committed.
   • Configuration Consistency:
     • Use the centralized config/config.js for accessing environment variables.

8. Test the Application

   • Functionality Tests:
     • Run the server and test all endpoints to ensure they work as expected.
     • Verify that rate limiting and error handling are functioning correctly.
   • Regression Testing:
     • Ensure that existing features are not broken by the refactoring.

---

Acceptance Criteria

• The backend code is reorganized according to the new modular structure.
• All functionalities work as before after the refactoring.
• The codebase is more maintainable, readable, and scalable.
• No sensitive information is committed to the repository.
• All tests pass successfully, and the application runs without errors.

---

Additional Notes

• Dependencies:

  • Ensure all necessary npm packages are installed after restructuring.
  • Update package.json if needed.

• Code Style:

  • Follow consistent coding conventions and linting rules throughout the codebase.

• Documentation:

  • Update the README file to reflect the new project structure and provide setup instructions.
  • Comment code where necessary to explain complex logic.

• Collaboration:

  • Coordinate with team members to ensure everyone is aware of the changes.
  • Merge changes into the main branch carefully to avoid conflicts.

---

Reference: Complete Code Implementation

Please refer to the attached code snippets for the complete implementation of the new project structure:

1. Environment Variables:

   • backend/.env.example

2. Configuration:

   • backend/config/config.js

3. Server Setup:

   • backend/server.js

4. Middleware:

   • backend/middleware/rateLimiter.js
   • backend/middleware/authMiddleware.js
   • backend/middleware/errorHandler.js

5. Routes:

   • backend/routes/index.js
   • backend/routes/spotifyRoutes.js
   • backend/routes/authRoutes.js

6. Controllers:

   • backend/controllers/spotifyController.js
   • backend/controllers/authController.js

7. Services:

   • backend/services/spotifyService.js
   • backend/services/spotifyClientCredentials.js
   • backend/services/authService.js

8. Utilities:

   • backend/utils/logger.js

9. Models:

   • backend/models/user.js (optional for future use)

---

Example: backend/server.js

// server.js
const express = require('express');
const session = require('express-session');
const config = require('./config/config');
const routes = require('./routes');
const passport = require('passport');
const { globalLimiter } = require('./middleware/rateLimiter');
const errorHandler = require('./middleware/errorHandler');
require('./services/authService'); // Initialize passport strategies

const app = express();

// Apply global rate limiter
app.use(globalLimiter);

// Configure Express session
app.use(
  session({
    secret: config.sessionSecret,
    resave: false,
    saveUninitialized: false,
    cookie: {
      secure: config.nodeEnv === 'production', // Set to true in production
      httpOnly: true, // Helps prevent XSS
      maxAge: 24 * 60 * 60 * 1000, // 1 day
    },
  })
);

// Initialize Passport
app.use(passport.initialize());
app.use(passport.session());

// Use routes
app.use('/', routes);

// Error handling middleware
app.use(errorHandler);

// Start the server
app.listen(config.port, () => {
  console.log(`Server is running on http://localhost:${config.port}`);
});

---

Please refer to the full code snippets provided in the project repository for detailed implementation.

---

Action Required

• Developers: Please begin refactoring the backend codebase as per the outlined tasks.
• Reviewers: After implementation, thoroughly review the changes and test all functionalities.
• Project Manager: Monitor progress and ensure the refactoring aligns with project goals.

---

Thank you for your attention to this important improvement to our codebase.

[1varunvc]

The above implementation increased complexity. Use the following feature-based modularity for a rather readable code.

[1varunvc]

Refactor Backend Codebase for Feature-Based Modularity and Scalability

---

Description:

The current backend project structure needs to be refactored to improve readability, maintainability, and scalability. This refactor will involve modularizing the codebase by feature and making several other structural improvements to simplify future development and onboarding for new developers.

---

Acceptance Criteria:

1. Further Modularity by Feature:

   • Organize the backend directory into feature-based directories.
   • Create separate folders for each major feature, such as auth/, spotify/, and test/.
   • Place respective routes, controllers, services, and middleware within these feature-based directories.

2. Flatten Middleware and Utility Structure:

   • Move middleware that is specific to a feature (e.g., authMiddleware.js) into the respective feature directory.
   • Combine general utility files into fewer modules (e.g., combine related middleware such as rate limiter and error handler into utils/).
   • Remove unnecessary levels of nesting in the project structure.

3. Use Index Files for Imports:

   • Create index.js files in each feature and utility directory to export all related modules.
   • Update imports across the project to leverage these index files, reducing verbosity.

4. Document Folder Purpose:

   • Add README.md files in each folder to explain the purpose of the folder, general conventions, and any specific instructions.
   • The documentation should help new developers understand the structure quickly and make changes without confusion.

---

New Project Structure Example:

backend/
├── .env
├── .env.development
├── .env.production
├── package.json
├── server.js
├── config/
│   └── config.js
├── auth/
│   ├── index.js
│   ├── authRoutes.js
│   ├── authController.js
│   ├── authService.js
│   ├── authMiddleware.js
│   └── README.md
├── spotify/
│   ├── index.js
│   ├── spotifyRoutes.js
│   ├── spotifyController.js
│   ├── spotifyService.js
│   ├── spotifyClientCredentials.js
│   └── README.md
├── test/
│   ├── index.js
│   ├── testRoutes.js
│   ├── testController.js
│   └── README.md
├── models/
│   ├── user.js
│   └── README.md
├── utils/
│   ├── index.js
│   ├── rateLimiter.js
│   ├── errorHandler.js
│   ├── logger.js
│   └── README.md
└── README.md

---

Additional Details:

• The refactor should preserve all current functionality of the backend API.
• Ensure that the environment configurations are updated if required.
• All paths for importing modules must be updated to align with the new structure.
• This refactor aims to make the codebase easier to navigate, maintain, and scale as new features are added.

---

Tasks Breakdown:

1. Organize by Feature:

   • [ ] Move existing auth, spotify, and test code into new feature-specific directories.

2. Flatten Middleware and Utility Structure:

   • [ ] Move authMiddleware.js into the auth/ directory.
   • [ ] Combine utility modules such as rateLimiter.js and errorHandler.js into a single utils/ directory.

3. Create and Use Index Files:

   • [ ] Add index.js files in each feature directory and utility directory.
   • [ ] Update all imports to use these index.js files to simplify code.

4. Documentation:

   • [ ] Add README.md files in each feature directory explaining the folder’s purpose.
   • [ ] Update the root README.md to reflect the new structure.

---

Expected Impact:

• Improved Readability: Files grouped by feature will make it easier to understand and locate related functionality.
• Better Scalability: Feature-based modularity will simplify adding new features without causing confusion or over-cluttering existing directories.
• Enhanced Maintainability: Simplified imports and combined utilities will make the project easier to navigate and maintain over time.

---

Labels:

• refactoring
• enhancement
• backend
• modularity

---

Development Notes:

• Perform thorough testing to ensure all API endpoints function correctly after restructuring.
• Ensure new developers can understand the refactored codebase more easily through the added documentation.

---

Please assign this issue to the backend team and prioritize it for the upcoming sprint.