New algorithm logic

[harshitasrivastav28]

• We have aligned and presented the audience the new algorithm logic.
• Next step: Setting up job architecture and deployment pipeline

[harshitasrivastav28]

• GA user attributes are not flowing in big query - 1 day lag is there before data starts flowing

[harshitasrivastav28]

Issues identified: discrepancy in user attributes flowing as "NULL". Before and after login the user IDs are not getting merged

Reference doc: https://support.google.com/analytics/answer/9268042?sjid=12265376955839615970-AP

[harshitasrivastav28]

Implementation discussion is planned today

[harshitasrivastav28]

• Phase 2 issue to be created for hashtag scraping from youtube and adding it to reocomendation logic for user. - Understand if this can be done in current phase and can push hashtags to big query
• Connect with Komal and Ravi to create an issue around endpoint changes requires to achieve the above task

[Natasha-GB]

• Batch jobs and tables have been set
• Explore third party options
• Google's VectorDB option to be looked at
• Set up monitoring on DS services

[harshitasrivastav28]

• We will have to go with the third party.
• Google form for big query has been filled. Awaiting response for support on google cloud.
• Next steps:

1. Await response from google to get it done from big query
2. Another provided to spend time on pursuing that front and see if all operations are available there. Set the similar search script and see if the algorithm is returning the similar batch of videos.

[harshitasrivastav28]

Next step:

1. Whenever we get new data, we push that to upstash
2. Setting up sanity check for vectorDB sync
3. Nudging on google channel
4. Check if the required functionalities are in the database: https://cloud.google.com/alloydb/ai?hl=en (This is transactional database which is not there in big query)

[Natasha-GB]

• Can we get realtime location for users?
• Successful video play logic on canisters

[jay-dhanwant]

ML server setup

Logic (2 days)

• Setup upstash server
• Sync bigquery to upstash
• Heuristic checks for sync (can be picked later)
• Enable API authentication
• Finalising the logic

Deployment (1 day)

• adding github secret
• deploying to fly.io

Testing and Benchmarking (1-2 days post integration)

[Natasha-GB]

• Need to set up Upstash server for prod
• E2E for Staging to be verified on Jay's end

[harshitasrivastav28]

Next Step:

• Working on server setup today 1st half expected to be completed
• Test it with Komal then we can close it from DS end

[jay-dhanwant]

ML server setup

• [x] Setup upstash server & initialise
• [x] Sync bigquery to upstash
• [x] Heuristic checks for sync (can be picked later)
• [x] Enable API authentication
• [x] Complete the DS codebase & local testing (2 days)
• [x] Change & test the new logic once canister data is available
• [x] code review
• [x] Adhoc changes
• [x] Deployment (1 day)

Testing and Benchmarking (1-2 days) ~ to be picked after the tech integration is done from the backend and frontend

[harshitasrivastav28]

• Post cannister update Jay will need another 2 days and 1 additional day for testing

[Natasha-GB]

• Git repo initiation today

[harshitasrivastav28]

• ML server is working E2E and repo is shared.
• Start fetching likes
• Create a logic for sampling Monday Task: Deployment to Fly

[Natasha-GB]

• Final logic pending only

[harshitasrivastav28]

• Need 1-2 days to complete the ML side of things
• Rust side of things have been closed.

[Natasha-GB]

• Server setup done
• Issue: video uploaded by the importer, not getting reflected in storage
• Popularity integration (2-3 hours)
• Rest sorted

[harshitasrivastav28]

• Wrapped up ML feed server and analysed the behaviour
• Found some issue and noting in notion (DS issues to be taken up in next phase)
• Popularity needs to be toned down as NSFW video tagging is creating issue
• Not happening in main feed as its driven by freshness
• Staging link shared where we can see the new logic however feeded content is not getting tagged in the feed yet
• Initial caching as current load time is high for personalised feed

[Natasha-GB]

• Freshness is fetched
• Not using popularity due to NSFW tagging
• Build a clustering mechanism to identify NSFW (people need to help us tag so that we can identify them correctly), use open source models to generate annotations
• If NSFW toggle is on > show them initial training data
• If toggle is off > for cold-start > have videos under guardrails (guardrail - how? warm annotations)
• Cannot judge about user's preference in the first go, however, lots of NSFW complaints raised
• HON Game data to still be incorporated into the model
• Any negative response should be considered as a strong signal to not display NSFW videos
• Put FE on hotornot.wtf for data gathering

Next Steps >

• Upstash to pgvector to be explored
• Behaviour to be confirmed
• Instance sizing

[Natasha-GB]

• Test vector db offerings
• Finalise route for AI feed
• Option 1: Go as is (deployed to new URL - hotornot.wtf - use for internal testing/organic audience) - recommended for now
• Option 2: Hard Signals from user (Reflective and Predictive: blurring out videos > See Anyway; prevents exposure to all)
• Option 3: NSFW tags and take a business decision to not let these videos flow

[Natasha-GB]

• VectorDB: BQ pre-filtering and querying solving our criteria (okay to go forward with)
• Refactoring
• Freshness along with popularity and relevance to be incorporated

[Natasha-GB]

• Metadata in BQ flowing correctly
• Integration done
• Plan to ship today

[Natasha-GB]

• v0: freshness added to serve more HON games and newer videos seeded
• Testing done from Jay's end, signals make sense (tested locally)
• Deployment to staging to be done today

[harshitasrivastav28]

• Feed is not behaving as expected. Need to spend time to check what is not working fine
• Connect with Devansh to understand the next step on feed side

[Natasha-GB]

• Testing in local is in progress to figure out ongoing issues (Komal & Jay working together on this)
• Test containers (only off-chain pending and to be set up)

[harshitasrivastav28]

• Lets start building the ML feed phase 2
• Reports on feed to be part of the logic implementation of phase 2
• Prioritise it over other task.
• If there is any video not part of listing but is proposed by model, figure out a way to ensure it should not break and become part of the list

[harshitasrivastav28]

• Create an issue around removing the duplication videos from the feed

[harshitasrivastav28]

This has been closed from @jay-dhanwant end.