The goal is to reach feature parity with the existing JavaScript implementation. Below is a comprehensive checklist of features and functionalities that need to be implemented in pubky-nexus for feature parity. Please ensure that each feature is thoroughly tested and documented.
1. Profiles
[x] Implement user profile retrieval. Currently (GET /profile/:userId)
[x] Implement complete profile data retrieval with additional information (follower count, tags, etc.)
2. Graph (Social Graph) Management
[x] #32
[x] #57
[x] Implement recommended profiles retrieval. Currently (GET /graph/recommended-profiles/:userId). We will use most-followed for now as in skunk-works.
3. Notification System
[x] Implement notifications retrieval. Currently (GET /notifications/:userId)
[x] #31
[x] Handle various notification types and ensure accurate notification delivery
[x] Ensure pagination and cursor-based retrieval
4. Streams (Feeds and Timelines)
[x] #55
[x] Support various stream filters (reach, sorting, etc.)
[x] Ensure pagination and cursor-based retrieval
[x] Streams by tag
5. Tags and Tagging System
[x] Implement hot tags retrieval. Currently (GET /hot-tags)
[x] Implement user tag management (indexing and deletion of tags)
[x] Handle different types of tags (post tags, profile tags, etc.)
6. Bookmarks
[x] Implement bookmarks retrieval. Currently (GET /bookmarks/:userId)
[x] Implement bookmark indexing and deletion
[x] Ensure proper handling of bookmark data
7. Search
[x] #72
[x] Implement stream of user profiles that wrap user id search endpoint
8. Post Management
[x] #59
[x] Implement user feed retrieval. Currently (GET /posts/:authorId)
[x] #65
[x] Support post indexing, deletion, and interaction handling (replies, reposts, etc.)
[x] Implement popularity scoring and ranking for posts
9. Indexer and Database Management
[x] Implement a Rust-based indexer similar to the current JavaScript implementation with Redis
[x] Implement a Graph database for WoT inferences and to keep a full view of the latest state of the social network
[x] Set up graph schema and tables for users, posts, relationships
[x] Set up index schema for notifications, streams, tags, bookmarks, and posts (prefixes and keys)
[ ] Implement transaction handling and ensure data consistency as needed.
10. Files
[x] Implement handling of user static files, including ingestion, serving, etc.
[x] Implement processing files (like images) and providing extra functionality, like resizing.
11. Homeserver watcher and Populating the databases
[x] #99
[x] Ensure we use MERGE or any sort of check needed to avoid adding to the graph duplicates of existing content.
[ ] Handle the deletions, both for the graph and for the index keys, sets and sorted sets.
[ ] Ensure if we get a repeated event we throw/catch an error and avoid triggering duplicate notifications.
12. Performance
[x] Implement Criterion for benchmarking
[x] Optimize performance for high concurrency and low latency
13. Testing and Documentation
[x] Implement integration testing practices to ensure end-to-end functionality
[x] Optionally also develop comprehensive unit tests for all modules and endpoints
[x] Document all APIs, data structures, and expected behaviors
[x] Optionally provide migration code, guides and documentation for the new Rust implementation
Additional Considerations:
[x] Indexer API will break with the current /mvp API version.
[x] Provide support for transition for the JavaScript frontend team. The .social part of the SDK will not exist anymore and all social functionality will move into pubky-app repository using OpenAPI standard.
Principles
Maximum normalization
We have to define the minimum atomic units that will be served. These are easy to define: each Neo4J graph node or relationship is the minimal unit. Then define a struct REUSE and REUSE the same. Do not re-invent the same objects.
Cache in index everything you can
Every type we implement should be able to myObjec.set_index() and MyStruct:try_from_index() by using our custom RedisOps trait.
tipogi
commented
3 months ago
The goal is to reach feature parity with the existing JavaScript implementation. Below is a comprehensive checklist of features and functionalities that need to be implemented in
pubky-nexusfor feature parity. Please ensure that each feature is thoroughly tested and documented.1. Profiles
GET /profile/:userId)2. Graph (Social Graph) Management
GET /graph/recommended-profiles/:userId). We will usemost-followedfor now as in skunk-works.3. Notification System
GET /notifications/:userId)4. Streams (Feeds and Timelines)
5. Tags and Tagging System
GET /hot-tags)6. Bookmarks
GET /bookmarks/:userId)7. Search
8. Post Management
GET /posts/:authorId)9. Indexer and Database Management
10. Files
11. Homeserver watcher and Populating the databases
MERGEor any sort of check needed to avoid adding to the graph duplicates of existing content.12. Performance
13. Testing and Documentation
Additional Considerations:
/mvpAPI version.[x] Provide support for transition for the JavaScript frontend team. The
.socialpart of the SDK will not exist anymore and all social functionality will move intopubky-apprepository using OpenAPI standard.Principles
Maximum normalization
We have to define the minimum atomic units that will be served. These are easy to define: each Neo4J graph node or relationship is the minimal unit. Then define a struct REUSE and REUSE the same. Do not re-invent the same objects.
Cache in index everything you can
Every type we implement should be able to
myObjec.set_index()andMyStruct:try_from_index()by using our customRedisOpstrait.