search

aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.42k stars 2.12k forks source link

OpenSearch and Amplify @searchable directive fails to deliver correct results for total and aggregation operations when the DynamoDB table has more than 10000 items #11086

Closed pyla96 closed 1 year ago

pyla96 commented 1 year ago

Before opening, please confirm:

JavaScript Framework

React

Amplify APIs

GraphQL API

Amplify Categories

api

Environment information

``` System: OS: Windows 10 10.0.19045 CPU: (12) x64 AMD Ryzen 5 3600 6-Core Processor Memory: 7.28 GB / 15.93 GB Binaries: Node: 14.17.5 - c:\program files\nodejs\node.EXE npm: 6.14.14 - c:\program files\nodejs\npm.CMD Browsers: Edge: Spartan (44.19041.1266.0), Chromium (110.0.1587.69) Internet Explorer: 11.0.19041.1566 npmPackages: @aws-amplify/cli: ^5.2.1 => 5.2.1 @aws-amplify/ui-components: ^1.9.6 => 1.9.6 (1.7.2) @aws-amplify/ui-react: ^2.20.0 => 2.20.0 @aws-amplify/ui-react-internal: undefined () @aws-amplify/ui-react-legacy: undefined () @emotion/react: ^11.10.4 => 11.10.4 @emotion/styled: ^11.10.4 => 11.10.4 @fortawesome/fontawesome-svg-core: ^1.2.35 => 1.2.35 @fortawesome/free-brands-svg-icons: ^5.15.3 => 5.15.3 @fortawesome/free-regular-svg-icons: ^5.15.3 => 5.15.3 @fortawesome/free-solid-svg-icons: ^5.15.3 => 5.15.3 @fortawesome/react-fontawesome: ^0.1.14 => 0.1.14 @mui/icons-material: ^5.0.1 => 5.0.4 @mui/lab: ^5.0.0-alpha.49 => 5.0.0-alpha.51 @mui/material: ^5.10.3 => 5.10.3 @mui/styles: ^5.8.0 => 5.8.0 @mui/x-data-grid: ^5.16.0 => 5.16.0 @testing-library/jest-dom: ^5.11.4 => 5.14.1 @testing-library/react: ^11.1.0 => 11.2.7 @testing-library/user-event: ^12.1.10 => 12.8.3 @typeform/embed-react: ^1.21.0 => 1.21.0 @vanillaes/csv: ^3.0.1 => 3.0.1 amplify-ui-components-loader: undefined () apexcharts: ^3.27.3 => 3.27.3 aws-amplify: ^4.3.24 => 4.3.24 axios: ^0.21.4 => 0.21.4 bootstrap: ^5.0.2 => 5.0.2 chart.js: ^3.8.0 => 3.8.0 chart.js-auto: undefined () chart.js-helpers: undefined () date-fns: ^2.25.0 => 2.25.0 js-file-download: ^0.4.12 => 0.4.12 new-plugin-package: 1.0.0 react: ^17.0.2 => 17.0.2 (16.14.0, 15.7.0) react-apexcharts: ^1.3.9 => 1.3.9 react-bootstrap: ^2.0.0-beta.5 => 2.0.0-beta.5 react-bootstrap/AbstractModalHeader: undefined () react-bootstrap/Accordion: undefined () react-bootstrap/AccordionBody: undefined () react-bootstrap/AccordionButton: undefined () react-bootstrap/AccordionCollapse: undefined () react-bootstrap/AccordionContext: undefined () react-bootstrap/AccordionHeader: undefined () react-bootstrap/AccordionItem: undefined () react-bootstrap/AccordionItemContext: undefined () react-bootstrap/Alert: undefined () react-bootstrap/Anchor: undefined () react-bootstrap/Badge: undefined () react-bootstrap/BootstrapModalManager: undefined () react-bootstrap/Breadcrumb: undefined () react-bootstrap/BreadcrumbItem: undefined () react-bootstrap/Button: undefined () react-bootstrap/ButtonGroup: undefined () react-bootstrap/ButtonToolbar: undefined () react-bootstrap/Card: undefined () react-bootstrap/CardColumns: undefined () react-bootstrap/CardGroup: undefined () react-bootstrap/CardHeader: undefined () react-bootstrap/CardHeaderContext: undefined () react-bootstrap/CardImg: undefined () react-bootstrap/Carousel: undefined () react-bootstrap/CarouselCaption: undefined () react-bootstrap/CarouselItem: undefined () react-bootstrap/CloseButton: undefined () react-bootstrap/Col: undefined () react-bootstrap/Collapse: undefined () react-bootstrap/Container: undefined () react-bootstrap/Dropdown: undefined () react-bootstrap/DropdownButton: undefined () react-bootstrap/DropdownContext: undefined () react-bootstrap/DropdownItem: undefined () react-bootstrap/DropdownMenu: undefined () react-bootstrap/DropdownToggle: undefined () react-bootstrap/ElementChildren: undefined () react-bootstrap/Fade: undefined () react-bootstrap/Feedback: undefined () react-bootstrap/Figure: undefined () react-bootstrap/FigureCaption: undefined () react-bootstrap/FigureImage: undefined () react-bootstrap/FloatingLabel: undefined () react-bootstrap/Form: undefined () react-bootstrap/FormCheck: undefined () react-bootstrap/FormCheckInput: undefined () react-bootstrap/FormCheckLabel: undefined () react-bootstrap/FormContext: undefined () react-bootstrap/FormControl: undefined () react-bootstrap/FormFloating: undefined () react-bootstrap/FormGroup: undefined () react-bootstrap/FormLabel: undefined () react-bootstrap/FormRange: undefined () react-bootstrap/FormSelect: undefined () react-bootstrap/FormText: undefined () react-bootstrap/Image: undefined () react-bootstrap/InputGroup: undefined () react-bootstrap/InputGroupContext: undefined () react-bootstrap/ListGroup: undefined () react-bootstrap/ListGroupItem: undefined () react-bootstrap/Modal: undefined () react-bootstrap/ModalBody: undefined () react-bootstrap/ModalContext: undefined () react-bootstrap/ModalDialog: undefined () react-bootstrap/ModalFooter: undefined () react-bootstrap/ModalHeader: undefined () react-bootstrap/ModalTitle: undefined () react-bootstrap/Nav: undefined () react-bootstrap/NavContext: undefined () react-bootstrap/NavDropdown: undefined () react-bootstrap/NavItem: undefined () react-bootstrap/NavLink: undefined () react-bootstrap/Navbar: undefined () react-bootstrap/NavbarBrand: undefined () react-bootstrap/NavbarCollapse: undefined () react-bootstrap/NavbarContext: undefined () react-bootstrap/NavbarToggle: undefined () react-bootstrap/Offcanvas: undefined () react-bootstrap/OffcanvasBody: undefined () react-bootstrap/OffcanvasHeader: undefined () react-bootstrap/OffcanvasTitle: undefined () react-bootstrap/OffcanvasToggling: undefined () react-bootstrap/Overlay: undefined () react-bootstrap/OverlayTrigger: undefined () react-bootstrap/PageItem: undefined () react-bootstrap/Pagination: undefined () react-bootstrap/Popover: undefined () react-bootstrap/PopoverBody: undefined () react-bootstrap/PopoverHeader: undefined () react-bootstrap/ProgressBar: undefined () react-bootstrap/Ratio: undefined () react-bootstrap/Row: undefined () react-bootstrap/Spinner: undefined () react-bootstrap/SplitButton: undefined () react-bootstrap/Switch: undefined () react-bootstrap/Tab: undefined () react-bootstrap/TabContainer: undefined () react-bootstrap/TabContent: undefined () react-bootstrap/TabPane: undefined () react-bootstrap/Table: undefined () react-bootstrap/Tabs: undefined () react-bootstrap/ThemeProvider: undefined () react-bootstrap/Toast: undefined () react-bootstrap/ToastBody: undefined () react-bootstrap/ToastContainer: undefined () react-bootstrap/ToastContext: undefined () react-bootstrap/ToastHeader: undefined () react-bootstrap/ToggleButton: undefined () react-bootstrap/ToggleButtonGroup: undefined () react-bootstrap/Tooltip: undefined () react-bootstrap/TransitionWrapper: undefined () react-bootstrap/createChainedFunction: undefined () react-bootstrap/createWithBsPrefix: undefined () react-bootstrap/divWithClassName: undefined () react-bootstrap/getTabTransitionComponent: undefined () react-bootstrap/helpers: undefined () react-bootstrap/safeFindDOMNode: undefined () react-bootstrap/transitionEndListener: undefined () react-bootstrap/triggerBrowserReflow: undefined () react-bootstrap/types: undefined () react-bootstrap/useOverlayOffset: undefined () react-bootstrap/useWrappedRefWithWarning: undefined () react-chartjs-2: ^4.3.1 => 4.3.1 react-date-range: ^1.4.0 => 1.4.0 react-dom: ^17.0.2 => 17.0.2 (16.14.0, 15.7.0) react-dropzone-uploader: ^2.11.0 => 2.11.0 react-gauge-chart: ^0.4.0 => 0.4.0 react-nestable: 1.2.2 => 1.2.2 react-perfect-scrollbar: ^1.5.8 => 1.5.8 react-phone-input-2: ^2.14.0 => 2.14.0 react-redux: ^7.2.4 => 7.2.4 react-router-dom: ^5.2.0 => 5.2.0 (4.3.1) react-scripts: 4.0.3 => 4.0.3 redux: ^4.1.1 => 4.1.1 redux-thunk: ^2.3.0 => 2.3.0 retry: ^0.13.1 => 0.13.1 (0.12.0) utf8: ^3.0.0 => 3.0.0 xlsx: ^0.17.3 => 0.17.3 npmGlobalPackages: @aws-amplify/cli: 10.8.1 env-cmd: 10.1.0 ```

Describe the bug

When trying to retrieve the total count of items in a table of more than 10,000 items, the "total" field returns the incorrect result of 10,000 items.

Pagination won't work either because using the nextToken returned by the previous query also returns the value of 10,000.

This behavior also occurs for the sum operation.

Expected behavior

Either the total/sum operations return the correct result or pagination returns the rest of the results in the subsequent calls.

Reproduction steps

  1. create a dynamodb table in the graphql schema and add the searchable directive
  2. populate the table with more than 10000 items
  3. use the query created by searchable and get the total item count 
    query MyQuery {
searchTodos {
total
nextToken
}
}
  4. see the incorrect result

Code Snippet

// Put your code below this line.

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

chrisbonifacio commented 1 year ago

Hi @pyla96 👋

Since ElasticSearch v7, track_total_hits is false by default. In order to get an accurate total from OpenSearch you need to override the resolver and set track_total_hits to true in the search request.

pyla96 commented 1 year ago

Nice! that worked well, I couldn't find that information anywhere...