Improve Tag Management

[ajefts]

Currently, we have a tags api and a page to manage tags in the admin tool.

We have a need to keep these tags in sync with technologies and platforms that are associated to challenges and stored in the informix db. Eventually we'll deprecate the old tables, but for now we need them.

So, ...

In the admin tool, we need to make sure we update the tcs_catalog.technology_type and tcs_catalog.project_platform_lu tables whenever we add/edit/delete tags. This will keep the 2 data sets in sync.

In the tags db, we should leverage the concept of domains to segment technologies and platforms. Let's chat about the best way to do this.

[ajefts]

@huangqun Can you take this one?

[huangqun]

@ajefts sure, I can take it. Hopefully this is not too urgent though? Will have limited time until early next week because of Chinese New Year.

[ajefts]

yes, that's fine. Thanks.

On Thu, Jan 26, 2017 at 11:26 AM, huangqun notifications@github.com wrote:

@ajefts https://github.com/ajefts sure, I can take it. Hopefully this is not too urgent though? Will have limited time until early next week because of Chinese New Year.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/appirio-tech/admin-app/issues/48#issuecomment-275434457, or mute the thread https://github.com/notifications/unsubscribe-auth/ADorrqr6s_c_jLHu9g5ViMpkeNM4Y_eTks5rWMk2gaJpZM4LuA_F .

[huangqun]

A couple questions on this:

1. If there are existing tags previously created via the tags api, do we need to sync these to the informix databases too?

2. I guess this will be one way sync, which means we only sync into the informix database but NOT vice versa?

3. Not sure if I understand the domains part, if you have more details it will be very helpful.

Thanks!

[ajefts]

1. Yes, but that can just be a 1 time sync. We could do an export of the current tags and then generate the inserts for the informix tables.

2. After the initial sync, the idea is that the admin tool will keep both data sources in sync.

3. The idea with the domains is that we can use them to segment different types of tags. Currently, the only domain we have set is "SKILLS". I'm thinking we can add "TECHNOLOGY" and "PLATFORM" as new domains to store the Informix technologies and platforms.

[huangqun]

For 3: so this is about syncing the informix technologies and platforms to the tags db via the tags API, right? I think I misunderstood to be the other way and that's why I got confused.

[ajefts]

Yes, that's correct. Eventually, we want the tags api to be the one and only place for it. So, this is a step towards that goal.

[huangqun]

@ajefts do we already have API to update the tcs_catalog.technology_type and tcs_catalog.project_platform_lu tables? If not we'll need to build such APIs because we can't access the database directly from the admin-app.

And if we need to build such APIs, which service should we add the APIs to?

[ajefts]

We don't have those yet, but that was actually on @skyhit list for the challenge api: https://github.com/appirio-tech/ap-challenge-microservice/issues/89

@skyhit @huangqun If you guys want to split this work up between you in a way that makes more sense, I'm totally fine with that...

[huangqun]

Makes more sense for @skyhit to handle it now that he already managed most of the api. :-)

[huangqun]

@ajefts so I guess I should wait for the v3 api to be done and should not use the v2 api for technology types and platforms?

[ajefts]

Yes, I think so.

[huangqun]

@skyhit are the tag services done yet?

[ajefts]

@huangqun We now have the technologies and platforms endpoints. You can see the swagger at https://github.com/appirio-tech/ap-challenge-microservice/blob/1.1.4/swagger.yaml

thanks

[huangqun]

@ajefts cool, is there any test environment for members? If not is there a guide to deploy this locally so we can implement the features in admin-app and test it locally?

[ajefts]

Good question.

@skyhit Any thoughts on this? Do we have any mocks for the challenge service yet? Could members generate their own mock server using swagger editor?

[skyhit]

@ajefts @huangqun we didn't have mocks for the challenge services yet, but I think we can setup the services using docker approach.

especially for technologies and platforms endpoints, there are data in the informix database.

[appirio-jenkins]

Is that something you want to launch a challenge for?

On Fri, Feb 17, 2017 at 10:20 AM, skyhit notifications@github.com wrote:

@ajefts https://github.com/ajefts @huangqun https://github.com/huangqun we didn't have mocks for the challenge services yet, but I think we can setup the services using docker approach.

especially for technologies and platforms endpoints, there are data in the informix database.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/appirio-tech/admin-app/issues/48#issuecomment-280677953, or mute the thread https://github.com/notifications/unsubscribe-auth/AMs6-ASKNLFcJdhQnlE9Gi8XosVOJBEOks5rdbq2gaJpZM4LuA_F .

[huangqun]

@skyhit could you please point me to the tags api too? Is there any mock for this or is there a way to run it locally just like the technologies / platforms endpoints?

[ajefts]

@huangqun It's at https://github.com/appirio-tech/ap-tagging-microservice/blob/dev/swagger.yaml

I believe the readme has instructions for running locally.

[huangqun]

@ajefts regarding your previous comments:

Yes, but that can just be a 1 time sync. We could do an export of the current tags and then generate the inserts for the informix tables. ==> So it seems we only have "SKILLS" domain in existing tags service, so where should we sync these into the informix tables? We have technologies and platforms, not sure which should be used for skills?

Also if we sync to one of these tables, I believe it conflicts with creating two new domains ("TECHNOLOGY" and "PLATFORM")?

[ajefts]

Let's keep them separate. So, we'll have 3 domains - skills, technologies, and platforms. So, we'll basically be syncing informix techs/platforms with tags techs/platforms.

[huangqun]

Cool, that's the assumption I went with when launching the challenge. ;-)

[parthshah]

@ajefts there is a significant overlap between skills and technologies / platforms. How do you plan on syncing those?

[huangqun]

@ajefts Checking the database tables project_platform_lu and technology_types.

It seems that there are some fields missing in those tables so that syncing might have an issue.

project_platform_lu does not have status, categories, priority, and synonyms.

technology_types does not have categories, priority, synonyms, and the 4 audit fields.

How should we deal with these mismatches during the sync?

[huangqun]

@ajefts also it seems the tagging service only has a search endpoint, where are the create / delete / edit endpoints?

[parthshah]

@huangqun please look at https://github.com/appirio-tech/tc-tags-service ap-tagging-microservice was deprecated awhile ago.

On Thu, Feb 23, 2017 at 12:35 AM huangqun notifications@github.com wrote:

@ajefts https://github.com/ajefts also it seems the tagging service https://github.com/appirio-tech/ap-tagging-microservice/blob/dev/swagger.yaml only has a search endpoint, where are the create / delete / edit endpoints?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/appirio-tech/admin-app/issues/48#issuecomment-281928554, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5UfpQNFw4JLJiGFH1Nnm0YN4CdIHpiks5rfUS5gaJpZM4LuA_F .

[huangqun]

Thanks @parthshah

[skyhit]

@parthshah @ajefts if so, can we mark Deprecated in the repo title for clearness?:)

[ajefts]

@huangqun For the missing fields that don't have an exact mapping, we can use some defaults:

status: Approved categories: [TECHNOLOGY | PLATFORM ] priority: 0 synonyms: null audit fields: use current dates for create/modify calls. Use the current user for the audit user.

[ajefts]

@parthshah Good question.

@huangqun Could you do a diff between the current platforms/technologies and the current tags? I'd like to see a list of what we have that overlaps (i.e. A platform or tech is already in the tags list)

[huangqun]

@ajefts do you mean the tags / platforms / technologies data in production database?

[ajefts]

Yes, please. We have the endpoints for all 3 to see the prod data.

On Feb 24, 2017, at 11:47 PM, huangqun notifications@github.com wrote:

@ajefts do you mean the tags / platforms / technologies data in production database?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[huangqun]

@ajefts what's the endpoint for the tags api? I tried https://api.topcoder.com/v3/tags but it's not working.

[ajefts]

Put a / at the end.

On Feb 25, 2017, at 11:57 PM, huangqun notifications@github.com wrote:

@ajefts what's the endpoint for the tags api? I tried https://api.topcoder.com/v3/tags but it's not working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[huangqun]

OK, that works, I was able to get all tags, platforms and technologies from the production api. However there are quite a few records and I will need to write a small program to do the comparison...

[ajefts]

ok, please do. Or we could just copy them into excel and use formulas in there to compare....

On Sun, Feb 26, 2017 at 2:10 AM, huangqun notifications@github.com wrote:

OK, that works, I was able to get all tags, platforms and technologies from the production api. However there are quite a few records and I will need to write a small program to do the comparison...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/appirio-tech/admin-app/issues/48#issuecomment-282537917, or mute the thread https://github.com/notifications/unsubscribe-auth/ADorrk2QkdDJru9mB8ifvgaWd5Q-_O-cks5rgSV2gaJpZM4LuA_F .

[huangqun]

@ajefts please see the attached file for the duplicates I found: dups.txt

[ajefts]

thanks @huangqun

So, this list are items that match an existing "skill", right?

[huangqun]

Yes, all these also exist in the tags under "Skills" domain.

[appirio-jenkins]

Curious, is this list from an exact case insensitive string match, or does it take into account "synonyms" -- node.js vs NodeJS , Node JS (not sure if we have this exact case but just giving an example ?

On Wed, Mar 1, 2017 at 8:32 AM huangqun notifications@github.com wrote:

Yes, all these also exist in the tags under "Skills" domain.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/appirio-tech/admin-app/issues/48#issuecomment-283392074, or mute the thread https://github.com/notifications/unsubscribe-auth/AMs6-IlC69syFvrRWvP1DLKZNYyFexZBks5rhZ2IgaJpZM4LuA_F .

[huangqun]

That's just exact case insensitive string match.

[huangqun]

@ajefts @skyhit the platforms api doesn't seem to support deletion of platforms, is that intended?

public ApiResponse deletePlatform(@Auth AuthUser user, @PathParam("id") long id) throws SupplyException { return MetadataApiResponseFactory.createResponse("This is just a mock now"); }

This mean when we delete a tag from the PLATFORM domain, the deletion can't be synced.

[skyhit]

@ajefts @huangqun like for delete Technologies, we actually just mark it as deleted, the records is still there. for platform, there is no such field, so we didn't implement that

@ajefts what is the expected behavior for delete platforms, we can implement that.

[huangqun]

@ajefts did you see my latest comment here? https://github.com/topcoder-platform/admin-app/pull/56

In order to makes tags sync work correctly we need to add a new "informixId" field to the tags database.