GCCollab - Prototyping Backend

[LecCory]

I have created a few basic function app endpoints. The functions read and write to a Cosmos DB via the Gremlin API

URL: https://devgcx-gccollab-apim.azure-api.net/api/

• RegisterUser (method: POST | endpoint: gRegisterUser)

Required params passed in request body Vertex is auto timestamped and included as a property

{
    "firstName": "John",
    "lastName": "Doe",
    "username": "jdoe",
    "email": "jdoe@gmail.com",
    "mobile": "555-555-5555"
}

• GetUser (method: GET | endpoint: gGetUser)

Required params passed either in the body, or as query params

{
  "sourceUser":"jdoe"

output:
{
    "data": {
        "firstName": [
            "John"
        ],
        "lastName": [
            "Doe"
        ],
        "email": [
            "jdoe@gmail.com"
        ],
        "username": [
            "jdoe"
        ],
        "mobile": [
            "555-555-5555"
        ],
        "registeredAt": [
            1673289122960
        ],
        "status": [
            1 //Active
        ],
        "isAdmin": [
            0
        ]
    }
}
}

• CreateFriendConnection (method: POST | endpoint: gCreateFriendConnection)

Required params passed in request body

{
    "sourceUser":"jdoe",
    "targetUser":"janedoe"
}

• GetFriendList (method: GET | endpoint: gGetFriendList)

Required query params or in request body

{
    "sourceUser":"jdoe"
}

output:
{
    "data": [
        {
            "friendStatus": {
                "status": 1 //Requested not yet accepted by target
            },
            "friendInfo": {
                "username": [
                    "jadoe"
                ]
            }
        },
}

• CreateGroup (method: POST | endpoint: gCreateGroup)

Required params passed in request body Vertex is auto timestamped and included as a property The slug URL is created by passing the title into a helper function that does the conversion.

{
    "username":"jdoe", //name of the logged in user
    "title":"this is the name of my group",
    "summary":"this group is for testing purposes",
    "status": 1, //Active
    "pub":1 //Group is public 0 = private
}

Findings

While the handful of endpoints were written in node leveraging the gremlin library to communicate with the Cosmos back end. I came across a few issues.

1. The documentation for nodeJS implementations of gremlin is lacking at least from my research, so it makes it complicated when trying to find answers or solutions.
2. It appears that newer gremlin functions like elementMap() are not available in CosmosDB. This has resulted in longer more convoluted queries.
3. There's no way to enforce schema validation like you would in a mongoDB or SQL schema. Vertices only have 3 datatypes in CosmosDB (string, number, bool)
4. The queries read nicely, once they're written but coming up with them has been very difficult due to the lack of documentation and the complexity of what documentation is available.

It would be my suggestion as I personally have a little more experience with MongoDB and the Cosmos MongoDB API that we consider using MongoDB. We can still write graph queries, but we'd have better control over the schema and have access to better documentation. I feel that Gremlin has been properly evaluated, but I don't feel that it should be the solution we present to IMTD.

[vid]

From some quick research, MongoDB has a basic graph function, $graphLookup[1], but its performance is not comparable to a real graph database[2].

There are workarounds to the missing functions like elementMap[3], though they make queries more convuluted.

If we went with MongoDB, we would go with a well supported solution, but it would simply not be able to practically do things graph databases can, which is a central feature in a forward-looking social communication system, at least without writing a lot of code to flatten relationships. If we went with Gremlin, we would have to spend more time working around Microsoft's shortcomings in implementation and documentation, but we would have access to powerful graph functions.

Could we take a hybrid approach, where we use CosmosDB's Gremlin functionality for relationships, and its MongoDB functionality for everything else, using the same identifiers? Or would that be the same as managing two databases? I am not sure if we can access the same data using different query systems.

1. https://www.mongodb.com/databases/mongodb-graph-database
2. https://www.arangodb.com/2018/02/nosql-performance-benchmark-2018-mongodb-postgresql-orientdb-neo4j-arangodb/, https://medium.com/mongodb-performance-tuning/optimising-graph-lookups-in-mongodb-49483afb55c8
3. https://stackoverflow.com/questions/62589698/azure-cosmos-db-gremlin-elementmap

[vid]

We should also book an appointment with @microsoft, since we pay for them as a resource, to see if they will talk about any immediate plans for Gremlin support.

There are a few other options, such as using Cassandra, which is a big data query system which many systems build on, though I am not sure it's meant for real time queries.

[vid]

Here's an example of why we want good graph database features: You see a group of 20 people collaborating on a topic, don't know anyone. But want to know your closest connection to that group; "who do I know or who's in a group I've worked with that is closest to this topic." This is the kind of query graph databases excel at, but are impossible with databases like Mongo, even with its graph extension, after a few degrees. And we would want this kind of feature to work for 100k+ users.

To clarify, using a service approach, we can use Gremlin for the open source option, and ClippyShortest path for Azure. The important point would be to use consistent identifiers for the data, which also enables SEO and the larger ecosystem of Linked Data.

[vid]

Since we need search anyway, there is a backend for JanusGraph (a big data graph database) for ElasticSearch.

[LecCory]

Endpoints created, however these endpoints are not wired up to a database

Endpoint	Info
accept_post	accept_post params: user, guid, lang POST API authentication required No user authentication required
add_colleague	add_colleague params: profileemail, user, lang POST API authentication required No user authentication required
apply_post	apply_post params: user, message, guid, lang POST API authentication required No user authentication required
approve_colleague	approve_colleague params: profileemail, user, lang POST API authentication required No user authentication required
auth_gettoken	auth_gettoken params: username, password POST No API authentication required No user authentication required
create_opportinities1	create_opportinities1 params: user, formData, lang POST API authentication required No user authentication required
create_opportinities2	create_opportinities2 params: user, formData, lang POST API authentication required No user authentication required
create_opportinities3	create_opportinities3 params: user, formData, lang POST API authentication required No user authentication required
decline_colleague	decline_colleague params: profileemail, user, lang POST API authentication required No user authentication required
delete_post	delete_post params: user, guid, lang POST API authentication required No user authentication required
edit_wire	edit_wire params: user, message, guid, lang POST API authentication required No user authentication required
entity_url	fetch_entity_url params: user, guid, lang POST API authentication required No user authentication required
event_add_calendar	event_add_calendar params: user, guid, lang POST API authentication required No user authentication required
gc_invite	talent_cloud_invite params: email, en_message, fr_message POST API authentication required No user authentication required
get_blog	get_api_entity params: id GET No API authentication required No user authentication required
get_blogedit	get_blog_edit params: user, guid, lang POST API authentication required No user authentication required
get_blogpost	get_blogpost params: user, guid, lang POST API authentication required No user authentication required
get_blogposts	get_blogposts params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_blogpostsbycolleague	get_blogposts_by_colleague params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_blogpostsbycontainer	get_blogposts_by_container params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_blogpostsbyowner	get_blogposts_by_owner params: user, limit, offset, filters, lang, target POST API authentication required No user authentication required
get_bookmark	get_bookmark params: user, guid, lang POST API authentication required No user authentication required
get_bookmarks	get_bookmarks params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_bookmarksbyuser	get_bookmarks_by_owner params: user, limit, offset, filters, lang, target POST API authentication required No user authentication required
get_bookmarkscolleague	get_bookmarks_colleague params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_colleaguerequests	get_colleague_requests params: user, limit, offset, lang POST API authentication required No user authentication required
get_commentsall	get_comments_all params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_discussion	get_discussion params: user, guid, thread, lang POST API authentication required No user authentication required
get_discussionedit	get_discussion_edit params: user, guid, lang POST API authentication required No user authentication required
get_discussions	get_discussions params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_doc	get_doc params: user, guid, lang POST API authentication required No user authentication required
get_docs	get_docs params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_event	get_event params: user, guid, lang POST API authentication required No user authentication required
get_events	get_events params: user, from, to, limit, offset, lang POST API authentication required No user authentication required
get_eventsbycolleagues	get_events_by_colleagues params: user, from, to, limit, offset, lang POST API authentication required No user authentication required
get_eventsbyowner	get_events_by_owner params: user, target, from, to, limit, offset, lang POST API authentication required No user authentication required
get_fields	get_register_fields params: none GET No API authentication required No user authentication required
get_file	get_file params: user, guid, lang POST API authentication required No user authentication required
get_group	get_group params: user, guid, lang POST API authentication required No user authentication required
get_group_member_list	get_group_member_list params: group_guid, offset, limit, name GET No API authentication required No user authentication required
get_groupactivity	get_group_activity params: user, guid, limit, offset, lang, api_version POST API authentication required No user authentication required
get_groupblogs	get_group_blogs params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_groupdiscussions	get_group_discussions params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_groupdocs	get_group_docs params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_groupevents	get_group_events params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_groupfiles	get_group_files params: user, guid, limit, offset, lang POST API authentication required No user authentication required
get_groups	get_groups params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_members	get_members params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_memberscolleague	get_members_colleague params: profileemail, user, limit, offset, filters,lang POST API authentication required No user authentication required
get_message	get_message params: user, guid, thread, lang POST API authentication required No user authentication required
get_messages	get_messages params: user, limit, offset, lang POST API authentication required No user authentication required
get_messagescount	get_messages_count params: user, lang POST API authentication required No user authentication required
get_newsfeed	get_newsfeed params: user, limit, offset, lang POST API authentication required No user authentication required
get_notifications	get_notifications params: user, limit, offset, lang POST API authentication required No user authentication required
get_notificationscount	get_notifications_count params: user, lang POST API authentication required No user authentication required
get_opportunities	get_opportunities params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_opportunity	get_opportunity params: user, guid, lang POST API authentication required No user authentication required
get_profile	get_api_profile params: id GET No API authentication required No user authentication required
get_profile_by_gcid	get_api_profile_gcid params: gcid GET No API authentication required No user authentication required
get_seecalendar	get_see_calendar params: user, guid, lang POST API authentication required No user authentication required
get_sentmessages	get_sent_messages params: user, limit, offset, lang POST API authentication required No user authentication required
get_sentmessagescount	get_sent_messages_count params: user, lang POST API authentication required No user authentication required
get_skills	get_api_skillList params: count GET No API authentication required No user authentication required
get_user	get_user_data params: profileemail, user, lang POST API authentication required No user authentication required
get_useractivity	get_user_activity params: profileemail, user, limit, offset, lang,api_version POST API authentication required No user authentication required
get_userexists	get_user_exists params: user, lang POST API authentication required No user authentication required
get_usergroups	get_user_groups params: profileemail, user, lang POST API authentication required No user authentication required
get_wire	get_wire_posts params: query, limit GET No API authentication required No user authentication required
get_wirepost	get_wirepost params: user, guid, thread, lang POST API authentication required No user authentication required
get_wireposts	get_wireposts params: user, limit, offset, filters, lang POST API authentication required No user authentication required
get_wirepostsbycolleagues	get_wirepostsbycolleagues params: user, limit, offset, lang POST API authentication required No user authentication required
get_wirepostsbyuser	get_wirepostsbyuser params: profileemail, user, limit, offset, lang POST API authentication required No user authentication required
group_decline	decline_group_invite params: user, guid, lang POST API authentication required No user authentication required
group_invite	invite_group_member params: profileemail, user, guid, lang POST API authentication required No user authentication required
group_invitemembers	invite_group_members params: profileemail, user, guid, lang POST API authentication required No user authentication required
group_join	join_group_function params: user, guid, lang POST API authentication required No user authentication required
group_leave	leave_group_function params: user, guid, lang POST API authentication required No user authentication required
group_members	get_groups_members params: user, guid, limit, offset, lang POST API authentication required No user authentication required
group_stats	get_group_data params: type, group, lang GET No API authentication required No user authentication required
like_count	like_count params: guid, user, lang POST API authentication required No user authentication required
like_item	like_item params: user, guid, lang POST API authentication required No user authentication required
like_users	like_users params: guid, user, lang POST API authentication required No user authentication required
login_sso	login_sso params: email, sub, lang POST No API authentication required No user authentication required
login_user	login_user params: user, password, lang POST No API authentication required No user authentication required
login_userforchat	login_user_for_chat params: user, lang POST API authentication required No user authentication required
login_userfordocs	login_user_for_docs params: user, guid, lang POST API authentication required No user authentication required
login_userforurl	login_user_for_url params: user, url, lang POST API authentication required No user authentication required
member_stats	get_member_data params: type, lang GET No API authentication required No user authentication required
post_discussion	post_discussion params: user, title, message, container_guid,access, open, topic_guid, lang POST API authentication required No user authentication required
post_wire	post_wire params: user, message, image, lang POST API authentication required No user authentication required
profile_create	profileCreate params: data POST API authentication required No user authentication required
profile_get	get_api_profile params: id GET No API authentication required No user authentication required
profile_update	profileUpdate params: id, data POST API authentication required No user authentication required
query_posts	query_the_posts params: user, password, object, query, group,guid, limit, offset, from_date, to_date, lang POST No API authentication required No user authentication required
read_message	read_message params: user, guid, lang POST API authentication required No user authentication required
register_user	register_new_user params: userdata, lang POST No API authentication required No user authentication required
remove_colleague	remove_colleague params: profileemail, user, lang POST API authentication required No user authentication required
reply_message	reply_message params: user, message, guid, lang POST API authentication required No user authentication required
reply_wire	reply_wire params: user, message, guid, image, lang POST API authentication required No user authentication required
revoke_colleague	revoke_colleague params: profileemail, user, lang POST API authentication required No user authentication required
save_blog	save_blog params: user, title, excerpt, body, container_guid,blog_guid, comments, access, status, lang POST API authentication required No user authentication required
save_event	save_event params: user, title, body, startdate, starttime,enddate, endtime, venue, room, allday,web_conference, url, additionnal, fees,contact_checkbox, contact_text,contact_email_text, contact_phone_text,picker_language, container_guid, event_guid,comments, access, status, lang POST API authentication required No user authentication required
search_user_skills	search_user_api_skills params: skills GET No API authentication required No user authentication required
send_message	send_message params: user, touser, subject, message, lang POST API authentication required No user authentication required
share_post	share_post params: user, message, guid, lang POST API authentication required No user authentication required
site_stats	get_site_data params: type, lang GET No API authentication required No user authentication required
submit_comment	submit_comment params: user, guid, message, lang POST API authentication required No user authentication required
system_api_list	list_all_apis params: none GET No API authentication required No user authentication required
time_stats	get_time_data params: type, lang GET No API authentication required No user authentication required
withdraw_post	withdraw_post params: user, message, guid, lang POST API authentication required No user authentication required