Backend: Implement Data Processing for YouTube

[1varunvc]

1. Process and Normalize Fetched Data: Develop backend logic to process and normalize data fetched from YouTube and Spotify APIs. Extract necessary fields such as song title, artist, album art, duration, and any other relevant metadata.
2. Unified Data Format: Ensure that the data from both sources is formatted into a consistent structure for the frontend to consume seamlessly.
3. Error Handling: Implement error checks for incomplete or inconsistent data and handle exceptions gracefully.

[1varunvc]

Approach/Strategy for Data Processing

1. Process YouTube Search Results

a. Extract Relevant Data

From the YouTube Search API response, extract the following fields for each video:

• videoId
• title
• publishedAt (to derive videoYear)
• thumbnails (include default, medium, high)
• channelId
• channelTitle
• description

b. Enhance Classification Logic

Classify each video as 'Audio' or 'Video' using the following criteria:

• Normalize Text:

  • Convert both title and description to lowercase.
  • Remove special characters to ensure consistent keyword matching.

• Audio Criteria:

  1. Description Starts With:
     • Check if the description starts with 'provided to youtube by'.
  2. Title Contains Keywords AND the Uploader is an Official Artist:
     • a. Match Keywords:
     • Use regular expressions to identify keywords like 'official audio', 'audio', 'lyric video', 'lyrics' in the title.
     • b. Identify Official Artist Channels (OAC):
     • Use the channelId to determine if the channel is an Official Artist Channel.
     • This can be achieved by checking against a maintained list of OAC channel IDs or utilizing YouTube's API features.
     • c. Fallback Strategies if OAC Identification Fails:
     • i. Compare channelTitle with Artist Name:
       • Extract the artist's name from the title and compare it with the channelTitle.
     • ii. Known Production Houses:
       • Use a predefined list of known production houses from productionHouses.js to check if the channelTitle matches any known production house.
  • Decision Flow:
  • Step 1: If description starts with 'provided to youtube by', classify as 'Audio'.
  • Step 2: Else, check if title contains the specified keywords.
    • If a is false: Classify as 'Video'.
    • If a is true:
    • Check b:
      • If b is true: Classify as 'Audio'.
      • If b is false:
      • Check c:
        • If c is true: Classify as 'Audio'.
        • If c is false: Classify as 'Video'.

• Video Criteria:

  • If none of the audio criteria are met, classify the item as 'Video'.

c. Separate Results

Organize the extracted data into two arrays within youtube.tracks:

• audios: Contains all items classified as 'Audio'.
• videos: Contains all items classified as 'Video'.

d. Data Validation and Fallbacks

• Validate Data Fields:

  • Ensure all necessary fields are present before processing an item.
  • If critical fields like videoId or title are missing, skip the item and log a warning.

• Assign Default Values:

  • Use meaningful defaults for optional fields (e.g., 'Unknown Title', 'No Description Available').

• Error Handling:

  • Continue logging errors, warnings, and informational messages as done previously to aid in debugging and monitoring.

---

2. Data Structure Adjustments

Maintain a consistent JSON structure for both YouTube and future integrations like Spotify:

{
  "youtube": {
    "tracks": {
      "audios": [
        {
          "videoId": "",
          "title": "",
          "videoYear": "",
          "thumbnails": {
            "default": "",
            "medium": "",
            "high": ""
          },
          "channelTitle": "",
          "description": ""
        }
        // ... more audio items
      ],
      "videos": [
        {
          "videoId": "",
          "title": "",
          "videoYear": "",
          "thumbnails": {
            "default": "",
            "medium": "",
            "high": ""
          },
          "channelTitle": "",
          "description": ""
        }
        // ... more video items
      ]
    },
    "artists": [],
    "albums": [],
    "playlists": []
  },
  "spotify": {
    "tracks": {
      "audios": [],
      "videos": []
    },
    "artists": [],
    "albums": [],
    "playlists": []
  }
}

• Future Integration Placeholders:

  • Leave empty arrays for artists, albums, and playlists under both youtube and spotify to facilitate future integration without significant restructuring.

• Consistent Schema:

  • Ensure the data structure is consistent across different platforms to simplify front-end processing.

---

3. Fetching Additional Details on User Interaction

a. New Endpoint

Create a new API endpoint (e.g., /api/youtube/details) that accepts a videoId and fetches additional details when a user clicks on a search result.

b. Fetch High-Resolution Data

Use the YouTube Videos API to retrieve:

• High-Resolution Thumbnails or Artwork
• Video Duration

Use the [Return YouTube Dislike API](https://returnyoutubedislikeapi.com/) to get:

• viewCount
• likes
• dislikes

Note: Ensure to handle these external API calls efficiently to manage quota usage.

---

4. Square Thumbnails

a. Thumbnail Processing

Since YouTube provides rectangular thumbnails:

• Server-Side Cropping:

  • Use image processing libraries like Sharp for Node.js to crop or resize images into square formats.
  • Asynchronous Processing:
  • Ensure image processing is non-blocking to maintain server performance.
  • Consider processing images in a separate worker thread or service.

• Lazy Processing:

  • Process thumbnails on-demand when they are first requested rather than processing all at once.

b. Alternative Sources

• High-Resolution Thumbnails:
  • Check if the YouTube Videos API provides thumbnails that are closer to square dimensions to reduce processing overhead.

---

5. Format Counts Based on Region

Implement logic to format numerical counts (viewCount, likes, dislikes) based on the user's region:

• Region Detection:

  • Allow the front-end to send the user's locale or region in the request headers.
  • Until the front-end is developed, use a default value (e.g., 'IN' for India or 'US' for the United States) picked from the request object.

• Formatting Rules:

  • India:
  • Use Thousand (T), Lakh (L), Crore (Cr).
  • Other Regions:
  • Use Kilo (K), Million (M), Billion (B).

• Formatting Utility:

  • Create a utility function to handle the formatting logic, making it easy to maintain and extend.

• Implement Region Detection Middleware:

  • Create regionMiddleware.js to extract the region from the request headers and set a default value if not provided.

---

6. Optimize API Usage and Quota Management

a. Handle Rate Limiting Errors

• Exponential Backoff Strategy:
  • Implement exponential backoff when retrying API requests after receiving rate limiting errors.
  • Increase the wait time between retries incrementally (e.g., 1 second, 2 seconds, 4 seconds).

b. Rotate Between Multiple API Keys

• API Key Rotation:
  • Maintain a pool of YouTube API keys.
  • Rotate between these keys to distribute quota usage evenly.
  • Implement logic to switch to the next key when a quota limit is reached for the current key.

---

7. Use Appropriate Service Naming and Structure

a. Rename Services for Clarity

• Rename youtubeService to youtubeAPI.js:

  • Since this module handles interactions with external YouTube APIs, a name like youtubeAPI.js is more descriptive.

• Similarly Rename Other Services:

  • spotifyService to spotifyAPI.js.
  • authService to authAPI.js (if applicable).
  • searchService to searchAPI.js (if applicable).

b. Create Separate Data Processing Module

• Create youtubeDataProcessor.js:

  • This module will handle all data processing, classification, and formatting logic for YouTube data.

• Benefits:

  • Separation of concerns improves code maintainability and readability.
  • Makes it easier to test and debug individual components.

---

8. Prepare for Server-Side Caching with Redis

• Skip In-Memory Caching:

  • Since Redis will be used soon, avoid implementing in-memory caching to prevent redundant work.

• Plan for Redis Integration:

  • Design your caching logic to be compatible with Redis.
  • Abstract caching functions so that integrating Redis involves minimal changes.

• Cache Invalidation Strategies:

  • Determine appropriate time-to-live (TTL) values for cached data.
  • Implement strategies for cache invalidation to ensure data freshness.

---

9. Security and Compliance

• Sanitize User Inputs:

  • Implement input validation and sanitization to prevent injection attacks and ensure data integrity.

• Handle Sensitive Data Carefully:

  • Securely store API keys and secrets.
  • Avoid logging sensitive information.
  • Ensure that error messages do not expose sensitive details.

---

10. Documentation and Maintainability

a. Inline Documentation

• Add comments and JSDoc annotations to explain complex logic and function usage.

b. Modularize Code

• Break down code into reusable functions and modules to improve readability and maintainability.

c. Unit Testing

• Write unit tests for critical functions, especially data processing and classification logic, to catch errors early.

---

11. Enhanced Error Handling and Logging

• Logging Practices:

  • Continue logging errors, warnings, and informational messages as you have been doing.

• Structured Logging:

  • Use structured logging to include context (e.g., request IDs) in your log messages, aiding in debugging and tracing.

• Error Handling:

  • Ensure that all exceptions are caught and handled gracefully to prevent application crashes.

---

Files to Modify or Add

Modify:

1. youtubeAPI.js

   • Handle interactions with YouTube's Search and Videos APIs.
   • Include methods for fetching search results and additional video details.
   • Implement API key rotation and rate limiting strategies.
   • Fetch data from returnyoutubedislikeapi.com for view counts, likes, and dislikes.

2. youtubeController.js

   • Update to handle the new data structure with separate audios and videos.
   • Add methods for the new endpoint that fetches additional details.

3. youtubeRoutes.js

   • Define new routes for fetching additional video details (e.g., /api/youtube/details).

4. searchController.js (if applicable)

   • Adjust to incorporate the updated YouTube data processing logic.

Add:

1. youtubeDataProcessor.js

   • Handle all data processing, classification, and formatting logic for YouTube data.
   • Functions include:
     • Data extraction.
     • Classification into 'Audio' or 'Video'.
     • Data validation and fallback strategies.

2. Utility Modules:

   • formatter.js
     • Functions to format views, likes, and dislikes based on the user's region.
   • imageProcessor.js
     • Functions to process and crop thumbnails into square formats.
   • regionDetector.js
     • Functions to detect the user's region from request headers and set default values.
   • cache.js
     • Abstraction layer for caching logic, designed for easy integration with Redis.

3. Configuration Files:

   • productionHouses.js
     • A list of known production house channel names for accurate classification.

4. Middleware:

   • regionMiddleware.js
     • Middleware to extract the user's locale or region from request headers and set a default value if not provided.

5. Logging Enhancements:

   • Update existing logging to include contextual information for better traceability.

---

Conclusion

By integrating these refinements into your approach, you will:

• Enhance Classification Accuracy:

  • Focus on Official Artist Channels (OAC) and fallback strategies for better differentiation between 'Audio' and 'Video' content.

• Optimize API Usage:

  • Batch requests, use partial responses, implement exponential backoff, and rotate API keys to reduce API calls and conserve quota.

• Improve User Experience:

  • Accurate data and formatted counts provide a more engaging interface.

• Prepare for Future Growth:

  • Modular code and consistent data structures facilitate easier integration of new features and platforms.

• Ensure Security and Compliance:

  • Proper input validation and secure handling of sensitive data protect both users and your application.

• Maintainability:

  • Well-documented and modular codebase simplifies future development and onboarding of new developers.

---

Next Steps

1. Implement the Updated Data Processing Logic:

   • Update your services and controllers with the refined data extraction and classification logic.

2. Develop Utility Modules and Middleware:

   • Build the utility functions for formatting and region detection.
   • Implement regionMiddleware.js to handle region extraction from request headers.

3. Plan for Redis Integration:

   • Structure your caching code to be compatible with Redis for when you implement it.

4. Test Thoroughly:

   • Validate that all components work together and handle edge cases gracefully.
   • Ensure that region formatting works correctly with default values and when the front-end provides the region.

5. Enhance Error Handling and Logging:

   • Continue logging errors, warnings, and informational messages, ensuring that all logs are useful for debugging.

6. Rename Services Appropriately:

   • Rename youtubeService to youtubeAPI.js and similarly for other services to reflect their purpose better.

7. Create Separate Data Processing Module:

   • Implement youtubeDataProcessor.js to handle data processing separately from API interactions.

[1varunvc]

TODO: Consider implementing backend/utils/imageProcessor.js in the front-end instead.