search

topcoder-platform / topcoder-x-ui

The website and general UI of Topcoder-X
4 stars 16 forks source link

[$100] Update `scanOne` to query #256

Closed jmgasper closed 4 years ago

jmgasper commented 4 years ago

@afrisalyp - We narrowed down the problem with duplicates to us not implementing scanOne properly.

Scanning only scans a subset of larger tables at once, meaning we have to scan multiple times to scan the whole table, which we don't do. This explains why sometimes it randomly works (found in the first "page"), but other times doesn't (on a page that wasn't scanned).

We've set up indices on dev and prod that we can query directly, instead of using scanOne. Can you make the proper code updates for this please? This should fix the last big issue (with the duplicate issues being created).

image

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108050 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108051 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108052 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108053 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

[500]: Challenge for the updated issue 256 is creating, rescheduling this event

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

[500]: Challenge for the updated issue 256 is creating, rescheduling this event

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108054 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108054 has been updated - it has been assigned to afrisalyp.

This is an automated message for ghostar via Topcoder X

afrisalyp commented 4 years ago

Scanning only scans a subset of larger tables at once, meaning we have to scan multiple times to scan the whole table, which we don't do. This explains why sometimes it randomly works (found in the first "page"), but other times doesn't (on a page that wasn't scanned).

@jmgasper - Owh, that's weird behavior of dynamodb that I've never realized it. But it sounds good that we already know the real problem.

We've set up indices on dev and prod that we can query directly, instead of using scanOne. Can you make the proper code updates for this please? This should fix the last big issue (with the duplicate issues being created).

Yeah, I'm going to work on it. Thanks.

jmgasper commented 4 years ago

@afrisalyp - You're not the only one. Multiple reviewers and myself missed that as well, and because we don't have as much test data in dev as we do in prod, we never really saw the issue. At least we have an idea of what the issue is now!

afrisalyp commented 4 years ago

@jmgasper - Yeah, that's good.

Topcoder X uses scan for all data fetching, not only the Issue table. Do we need to change all of scan usage to query? Or we just focus on the Issue table?

jmgasper commented 4 years ago

Eventually we'll have to, but let's focus on Issue table for now, just to get past this blocking issue.

afrisalyp commented 4 years ago

Yeah. agreed. Project and UserMapping tables aren't ready for query yet. They don't have required indexes.

afrisalyp commented 4 years ago

Two PR for topcoder-x-processor and topcoder-x-ui submitted.

jmgasper commented 4 years ago

@afrisalyp - I'm not sure this is working properly for new tickets, where the query won't return anything:

debug: Enter queryOne.
debug: {"number":{"eq":258},"provider":{"eq":"github"},"repositoryId":{"eq":135514613}}
debug: queryOne. Error. TypeError: Cannot read property 'map' of undefined
error: Error happened in process
TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Cannot read property 'map' of undefined, stack=TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), logged=true
debug: queryOne. Error. TypeError: Cannot read property 'map' of undefined
error: Error happened in process
TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Cannot read property 'map' of undefined, stack=TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), logged=true
afrisalyp commented 4 years ago

That's weird again. It's the same error as seen in https://github.com/topcoder-platform/topcoder-x-ui/issues/245. TypeError: Cannot read property 'map' of undefined

I'm sure that I've successfully tested the code in my local dev. Btw, what's the version of node and npm using by the prod?

jmgasper commented 4 years ago

@afrisalyp - Looks like prod uses node 8.12.0

Not sure what NPM version is used.

afrisalyp commented 4 years ago

I'm not really sure if this has an effect, but I found that the model in the migration script has different type in labels. The value in migration script is Array and it is required, but in topcoder-x-ui and topcoder-x-processor we have [String] and not required.

https://github.com/topcoder-platform/topcoder-x-ui/blob/d2cf9f93b6f295dd82e6beb922238fa42f94e929/scripts/mongodb-migration/dynamodb/models/Issue.js#L50

https://github.com/topcoder-platform/topcoder-x-ui/blob/d2cf9f93b6f295dd82e6beb922238fa42f94e929/src/models/Issue.js#L50

https://github.com/topcoder-platform/topcoder-x-processor/blob/71d442d4889e35353cc0c4292165571a80ac51f9/models/Issue.js#L51

jmgasper commented 4 years ago

@afrisalyp - Not sure that would affect us here because this is for a new ticket, so not something that was migrated over.

jmgasper commented 4 years ago

We can try to change it though - do you want to test in dev first?

afrisalyp commented 4 years ago

I don't see this error https://github.com/topcoder-platform/topcoder-x-ui/issues/256#issuecomment-558466013 in my local dev. Is that appear in x.topcoder-dev.com?

Also, didn't the migration script use for the prod only?

jmgasper commented 4 years ago

Yes, the migration script was for prod only. We can attempt to adjust the code to account for that change in prod, if you want.

afrisalyp commented 4 years ago

@jmgasper I'm not really sure that it's the problem. Still investigating in my local db

afrisalyp commented 4 years ago

@jmgasper - Now I'm pretty sure that it's the problem. :D I dumped my local DB. Here is it looks like

        {
            "assignedAt": {
                "N": "1574461009424"
            },
            "title": {
                "S": "Simultan issue 3"
            },
            "body": {
                "S": "<p>3</p>\n"
            },
            "labels": {
                "SS": [
                    "tcx_Assigned"
                ]
            },
            "number": {
                "N": "11"
            },
            "challengeId": {
                "N": "30055718"
            },
            "provider": {
                "S": "github"
            },
            "prizes": {
                "NS": [
                    "13"
                ]
            },
            "repositoryId": {
                "N": "223172322"
            },
            "id": {
                "S": "e67b2d56-112e-4451-b75a-684126413c21-1574460996831"
            },
            "assignee": {
                "S": "telaga"
            },
            "projectId": {
                "S": "93aa952e-da9e-45b9-b920-88b2c0560d04-1574340401139"
            },
            "updatedAt": {
                "N": "1574461013506"
            },
            "status": {
                "S": "challenge_creation_successful"
            }
        }

And if we compare with the prod DB record that you just sent me

        {
            "projectId": {
                "S": "951d1d30-75c8-4b6f-8195-f0f4bd1c90d4-1574306750843"
            },
            "challengeId": {
                "N": "30085497"
            },
            "repositoryId": {
                "N": "10469433"
            },
            "status": {
                "S": "challenge_creation_successful"
            },
            "assignedAt": {
                "N": "1551860245570"
            },
            "number": {
                "N": "133"
            },
            "labels": {
                "S": "[\"tcx_Assigned\",\"tcx_FixAccepted\",\"tcx_Assigned\"]"
            },
            "updatedAt": {
                "N": "1551921403829"
            },
            "provider": {
                "S": "gitlab"
            },
            "id": {
                "S": "aada0c1c-c170-4074-9ba1-de6db1096266-1574309109010"
            },
            "title": {
                "S": "PRIORITY - Handle NDA"
            },
            "body": {
                "S": "<p>This ticket will update the NDA page to be wired into the service:</p>\n<p>http://174.129.220.35:8000/registration/developer/nda</p>\n<p>Once the user has agreed to the NDA"
            },
            "prizes": {
                "NS": [
                    "350"
                ]
            }
        },

We have different labels type. "S" and "SS". And if we take a look into the stacktrace file /topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36

         const set = json[`${type}S`];
         return Promise.all(set.map((v) => {

It's related to the "S".

jmgasper commented 4 years ago

@afrisalyp - I made the change in prod, but now the query isn't working properly: Note that below the repositoryId isn't matching, nor is the provider.

debug: Enter queryOne.
debug: {"number":{"eq":260},"provider":{"eq":"github"},"repositoryId":{"eq":135514613}}
debug: EXIT process
debug: output arguments
debug: undefined
debug: queryOne. Result.
debug:  projectId=951d1d30-75c8-4b6f-8195-f0f4bd1c90d4-1574306750843, challengeId=30086732, repositoryId=10469433, status=challenge_creation_successful, assignedAt=Wed Mar 20 2019 11:39:42 GMT+0000 (UTC), number=260, labels=[tcx_OpenForPickup], updatedAt=Tue Nov 26 2019 07:41:40 GMT+0000 (UTC), provider=gitlab, id=27dbd483-841b-405f-8d50-9bfd3e542554-1574309381278, title=Update left menu component, body=<p>@suppermancool - We updated the left menu component to be able to be driven by a secondary nav:</p>
<p>https://gitlab.com/herc-dev-prtl/project-resources-devportal/wikis/Setup-Secondary-Navigation</p>
<p>One thing that's been requested is the ability to render and show up to <em>5</em> separate levels.  Is that something you can add support for in the component please?</p>
<p>We have no UI designs for this, and I've asked for them, but I wouldn't expect to get any.</p>
, prizes=[200]
error: Error happened in process
Error: Issue 260 is already in challenge_creation_successful
afrisalyp commented 4 years ago

@jmgasper - Fixed.

according to https://dynamoosejs.com/api/query/#querylimitlimit. Query also has the limitation. So I added all() to make sure we get proper data.

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108189 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Contest https://www.topcoder.com/challenges/30108189 has been updated - it has been assigned to afrisalyp.

This is an automated message for ghostar via Topcoder X

jmgasper commented 4 years ago

Payment task has been updated: https://software.topcoder-dev.com/review/actions/ViewProjectDetails?pid=30108189

This is an automated message for ghostar via Topcoder X