Datastore unable to stop / clear when logging out

mks11 commented 9 months ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Next.js

Amplify APIs

DataStore

Amplify Version

v6 "aws-amplify": "^6.0.9", from package.json

Amplify Categories

auth, api

Backend

Amplify CLI

Environment information

``` # Put output below this line System: OS: macOS 13.0 CPU: (8) arm64 Apple M1 Pro Memory: 96.66 MB / 16.00 GB Shell: 5.8.1 - /bin/zsh Binaries: Node: 18.18.0 - ~/.nvm/versions/node/v18.18.0/bin/node Yarn: 1.22.18 - /usr/local/bin/yarn npm: 9.8.1 - ~/.nvm/versions/node/v18.18.0/bin/npm Watchman: 2023.12.04.00 - /opt/homebrew/bin/watchman Browsers: Chrome: 120.0.6099.216 Edge: 120.0.2210.144 Safari: 16.1 npmPackages: @ampproject/toolbox-optimizer: undefined () @aws-amplify/ui-react: ^6.0.7 => 6.0.7 @aws-amplify/ui-react-internal: undefined () @babel/core: undefined () @babel/runtime: 7.22.5 @edge-runtime/cookies: 4.0.2 @edge-runtime/ponyfill: 2.4.1 @edge-runtime/primitives: 4.0.2 @hapi/accept: undefined () @headlessui/react: ^1.7.17 => 1.7.17 @heroicons/react: ^2.0.18 => 2.1.1 @mswjs/interceptors: undefined () @napi-rs/triples: undefined () @next/font: undefined () @next/react-dev-overlay: undefined () @opentelemetry/api: undefined () @segment/ajv-human-errors: undefined () @svgr/webpack: ^8.1.0 => 8.1.0 @tailwindcss/typography: ^0.5.10 => 0.5.10 @types/lodash: ^4.14.199 => 4.14.202 @types/node: latest => 20.10.5 @types/react: latest => 18.2.46 @types/react-dom: latest => 18.2.18 @vercel/nft: undefined () @vercel/og: 0.5.15 acorn: undefined () amphtml-validator: undefined () anser: undefined () arg: undefined () assert: undefined () async-retry: undefined () async-sema: undefined () autoprefixer: latest => 10.4.16 aws-amplify: ^6.0.9 => 6.0.9 aws-amplify/adapter-core: undefined () aws-amplify/analytics: undefined () aws-amplify/analytics/kinesis: undefined () aws-amplify/analytics/kinesis-firehose: undefined () aws-amplify/analytics/personalize: undefined () aws-amplify/analytics/pinpoint: undefined () aws-amplify/api: undefined () aws-amplify/api/server: undefined () aws-amplify/auth: undefined () aws-amplify/auth/cognito: undefined () aws-amplify/auth/cognito/server: undefined () aws-amplify/auth/server: undefined () aws-amplify/datastore: undefined () aws-amplify/in-app-messaging: undefined () aws-amplify/in-app-messaging/pinpoint: undefined () aws-amplify/push-notifications: undefined () aws-amplify/push-notifications/pinpoint: undefined () aws-amplify/storage: undefined () aws-amplify/storage/s3: undefined () aws-amplify/storage/s3/server: undefined () aws-amplify/storage/server: undefined () aws-amplify/utils: undefined () babel-packages: undefined () browserify-zlib: undefined () browserslist: undefined () buffer: undefined () bytes: undefined () ci-info: undefined () cli-select: undefined () client-only: 0.0.1 comment-json: undefined () compression: undefined () conf: undefined () constants-browserify: undefined () content-disposition: undefined () content-type: undefined () cookie: undefined () cross-spawn: undefined () crypto-browserify: undefined () css.escape: undefined () data-uri-to-buffer: undefined () date-fns: ^2.30.0 => 2.30.0 debug: undefined () devalue: undefined () domain-browser: undefined () edge-runtime: undefined () eslint: ^8.51.0 => 8.56.0 eslint-config-next: 13.5.4 => 13.5.4 eslint-plugin-import: ^2.28.1 => 2.29.1 events: undefined () find-cache-dir: undefined () find-up: undefined () fresh: undefined () get-orientation: undefined () glob: undefined () gzip-size: undefined () http-proxy: undefined () http-proxy-agent: undefined () https-browserify: undefined () https-proxy-agent: undefined () icss-utils: undefined () ignore-loader: undefined () image-size: undefined () is-animated: undefined () is-docker: undefined () is-wsl: undefined () jest-worker: undefined () json5: undefined () jsonwebtoken: undefined () loader-runner: undefined () loader-utils: undefined () lodash: ^4.17.21 => 4.17.21 lodash.curry: undefined () lru-cache: undefined () micromatch: undefined () mini-css-extract-plugin: undefined () nanoid: undefined () native-url: undefined () neo-async: undefined () next: latest => 14.0.4 node-fetch: undefined () node-html-parser: undefined () ora: undefined () os-browserify: undefined () p-limit: undefined () path-browserify: undefined () platform: undefined () postcss: latest => 8.4.32 (8.4.31) postcss-flexbugs-fixes: undefined () postcss-modules-extract-imports: undefined () postcss-modules-local-by-default: undefined () postcss-modules-scope: undefined () postcss-modules-values: undefined () postcss-preset-env: undefined () postcss-safe-parser: undefined () postcss-scss: undefined () postcss-value-parser: undefined () process: undefined () punycode: undefined () querystring-es3: undefined () raw-body: undefined () react: latest => 18.2.0 react-builtin: undefined () react-dom: latest => 18.2.0 react-dom-builtin: undefined () react-dom-experimental-builtin: undefined () react-experimental-builtin: undefined () react-horizontal-scrolling-menu: ^4.1.1 => 4.1.1 react-is: 18.2.0 react-refresh: 0.12.0 react-server-dom-turbopack-builtin: undefined () react-server-dom-turbopack-experimental-builtin: undefined () react-server-dom-webpack-builtin: undefined () react-server-dom-webpack-experimental-builtin: undefined () react-top-loading-bar: ^2.3.1 => 2.3.1 regenerator-runtime: 0.13.4 sass-loader: undefined () scheduler-builtin: undefined () scheduler-experimental-builtin: undefined () schema-utils: undefined () semver: undefined () send: undefined () server-only: 0.0.1 setimmediate: undefined () shell-quote: undefined () source-map: undefined () stacktrace-parser: undefined () stream-browserify: undefined () stream-http: undefined () string-hash: undefined () string_decoder: undefined () strip-ansi: undefined () superstruct: undefined () tailwindcss: latest => 3.4.0 tar: undefined () terser: undefined () text-table: undefined () timers-browserify: undefined () tty-browserify: undefined () typescript: latest => 5.3.3 ua-parser-js: undefined () unistore: undefined () util: undefined () vm-browserify: undefined () watchpack: undefined () web-vitals: undefined () webpack: undefined () webpack-sources: undefined () ws: undefined () xlsx: ^0.18.5 => 0.18.5 zod: undefined () npmGlobalPackages: corepack: 0.19.0 npm: 9.8.1 ```

Describe the bug

DataStore fails to clear or stop after logging out. This is the screenshot of the error error

This is the snippet from a top level layout

Component A

  Hub.listen("auth", async (data) => {
    switch (data.payload.event) {
      case "signedIn":
        console.log("user signed in");
        break;
      case "signedOut":
        setDSReady(false);
        console.log("user signed out");
        await DataStore.stop();
        await DataStore.clear();
        break;
    }
  });

  Hub.listen("datastore", async (data) => {
    console.log(data.payload.event, data.payload.data);
    if (data.payload.event === "ready") {
      setDSReady(true); // <-- this is to pass down the context from DataStore to children below.
    }
  });

And this the snippet of the code that should kick off DataStore in a component that is rendered after the user has signed up successfully

Component B

  useEffect(() => {
    fetchUser();
    if (user?.userId) {
      (async () => {
        await DataStore.start();
      })();
    }
  }, [user?.userId]);

fetchUser is just fetchUserAttributes() from Amplify v6.

Expected behavior

The DataStore should clear after user has signout without throwing the error.

Reproduction steps

Get a bare minimum NextJS Amplify app running
Add auth
Possibly add two components like the above

Code Snippet

Component A


  Hub.listen("auth", async (data) => {
    switch (data.payload.event) {
      case "signedIn":
        console.log("user signed in");
        break;
      case "signedOut":
        setDSReady(false);
        console.log("user signed out");
        await DataStore.stop();
        await DataStore.clear();
        break;
    }
  });

  Hub.listen("datastore", async (data) => {
    console.log(data.payload.event, data.payload.data);
    if (data.payload.event === "ready") {
      setDSReady(true); // <-- this is to pass down the context from DataStore to children below.
    }
  });

Component B

  useEffect(() => {
    fetchUser();
    if (user?.userId) {
      (async () => {
        await DataStore.start();
      })();
    }
  }, [user?.userId]);

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

cwomack commented 9 months ago

Hello, @mks11 👋. Sorry to hear you're experiencing this blocker. We're looking into this right now, but this could possibly be related to #12359 as well. That issue was experiencing similar problems when attempting to clear while an Auth event is happening.

Can you see if this comment from that issue (which also references 2 comments) helps at all? It details some steps to ensure that the Datastore.clear() has finished resolving before querying.

cwomack commented 9 months ago

@mks11, can you also help clarify where the Datastore.query() call is being made that is potentially throwing this error? I don't see it in the code snippets provided, but that may give us some insight as to why this is happening.

mks11 commented 9 months ago

Hi @cwomack, 👋 thank you so much for looking into it, this is the root layout for our NextJS application. removed some of the code.

If you check the bottom of the file, there is the observeQuery snippet. This is the subscriber that is inside a file following the convention of App Router from NextJS 13, like so src > app > (reports)>page.tsx, whereas the above part of the code (also commented) is at src > app > layout.tsx.

function Page() {
  const orgID = getOrgId(useUser());  // added snippet below
  const [reports, setReports] = useState<Report[]>([]);
  const router = useRouter();

  useEffect(() => {
    const sub = DataStore.observeQuery(
      Report,
      (r) => r.organizationID.eq(orgID),
      {
        sort: (s) => s.createdAt(SortDirection.DESCENDING),
      }
    ).subscribe(({ items }) => {
      setReports(items);
    });
    return () => sub.unsubscribe();
  }, [orgID]);

  return <div> (code removed) </div>
}

Amplify.configure(amplifyconfig, {
  ssr: true,
});

DataStore.configure({
  errorHandler: (err) => {
    console.warn("Datastore  err", err);
  },
  authModeStrategyType: AuthModeStrategyType.DEFAULT,
  conflictHandler: async (data: SyncConflict) => {
    console.log("@@@ conflict @@@ data", data);
    return DISCARD;
  },
  maxRecordsToSync: 50000,
});

function RootLayout({ children }: { children: React.ReactNode }) {
  const [isDSReady, setDSReady] = useState(false);
  Hub.listen("auth", async (data) => {
    switch (data.payload.event) {
      case "signedIn":
        console.log("user signed in");
        break;
      case "signedOut":
        setDSReady(false);
        await DataStore.stop();
        await DataStore.clear();
        console.log("user signed out");
        break;
    }
  });

  Hub.listen("datastore", async (data) => {
    console.log(data.payload.event, data.payload.data);
    if (data.payload.event === "ready") {
      setDSReady(true);
    }
  });

  return (
    <html>
      <body>
        <IsDSReadyProvider isDSReady={isDSReady}>
          <Authenticator hideSignUp>
            {({ user, signOut }) => {
              return (
                <PostSignIn user={user} signOut={signOut}>
                  {children}
                </PostSignIn>
              );
            }}
          </Authenticator>
        </IsDSReadyProvider>
      </body>
    </html>
  );
}

function PostSignIn({
  user,
  children,
  signOut,
}: {
  user: AuthUser | undefined;
  children: ReactNode;
  signOut: any;
}) {
  const segment = useSelectedLayoutSegment() || "";
  const [attrs, setAttrs] = useState<FetchUserAttributesOutput>();
  const [loading, setLoading] = useState(false);
  const [err, setErr] = useState<unknown>();
  const isDSReady = useContext(IsDSReady);

  useEffect(() => {
    fetchUser();
     (async () => {
       await DataStore.start();
     })();
  }, [user?.userId]);

  async function fetchUser() {
    try {
      setErr(undefined);
      setLoading(true);
      const attrs = await fetchUserAttributes();
      setAttrs(attrs);
    } catch (err) {
      setErr(err);
    } finally {
      setLoading(false);
    }
  }

  console.log("isDSReady", isDSReady);

  if (loading) {
    return (
           // code removed 
    );
  }

  if (!attrs?.name || !attrs?.email) {
    return;
  }

  if (!user || err) {
    console.log(err);
    return;
  }
  const _user = {
    userId: user.userId,
    name: attrs.name!,
    email: attrs.email,
    preferred_username: attrs.preferred_username,
  };

  return (
    <UserProvider user={_user}>
      <LoadingBar color="#6B83FF" progress={!isDSReady ? 30 : 100} />
      <aside>
          <button className="inline-flex" onClick={signOut}>
            <LogoutIcon fill={"#7E879C"} />
          </button>
      </aside>
      <div>
        {children}
      </div>
    </UserProvider>
  );
}

// context IsDSReady
export const IsDSReady = createContext<boolean>(false);

export function IsDSReadyProvider(props: {
  isDSReady: boolean;
  children: ReactNode;
}) {
  return (
    <IsDSReady.Provider value={props.isDSReady}>
      {props.children}
    </IsDSReady.Provider>
  );
}

// context UserProvider
export function UserProvider({
  user,
  children,
}) {
      <UserContext.Provider value={user}>
        <DispatchContext.Provider value={() => {}}>
          {children}
        </DispatchContext.Provider>
      </UserContext.Provider>
}

// here is another file where we are subscribing as our first view
function Page() {
  const orgID = getOrgId(useUser());  // added snippet below
  const [reports, setReports] = useState<Report[]>([]);
  const router = useRouter();

  useEffect(() => {
    const sub = DataStore.observeQuery(
      Report,
      (r) => r.organizationID.eq(orgID),
      {
        sort: (s) => s.createdAt(SortDirection.DESCENDING),
      }
    ).subscribe(({ items }) => {
      setReports(items);
    });
    return () => sub.unsubscribe();
  }, [orgID]);

  return <div> </div> // renders the view (code removed)
}

export function useUser() {
  const user = useContext(UserContext);
  return user!;
}
export const getOrgId = (user) => user.preferred_username;

stocaaro commented 9 months ago

Thank you for the detail about how your application uses DataStore. I tried to pull together a sample app that would use this code and encountered missing pieces. Would it be possible to share an application the is having this error as a public repo or share a private repo explicitely with @cwomack or myself?

Looking over this code, I have a couple questions:

Are you using datastore in the SSR context, or only client side? I ask because I don't think DataStore works server side. This doesn't appear to be related to your issue, but may be causing other issues.
It doesn't look like DataStore.query is called directly. I have checked and DataStore.observeQuery calls DataStore.query, which from the provided context must be where the error is coming from. How does your application ensure that the observeQuery calls are closed out and not re-activated while the stop and clear actions are in flight?

Answering questions is helpful, but more than anything getting this error reproduced will help us to provide guidance and fix any underlying issues more directly.

Thanks, Aaron

mks11 commented 9 months ago

@stocaaro Thank you Aaron for looking into it. Let me get back to you about sharing the repo. But let me quickly answer some of the questions

We are using DataStore only on the client side, all of the components including pages are marked as use client
We do unsubscribe the call within the useEffect (as shown in the code above), we do not have any extra measures to check this beyond. We use observeQuery in many places, and I think
- it'd be cumbersome to have this extra check.
- And when the user logs back in, we'll have to wait for Datastore to be ready (which in this scenario will just hang). Correct me if wrong, but I shouldn't have to wait for DataStore's "ready" event to start observing queries. We definitely want to show our users any data that is available at the time of login without waiting for any event that DataStore might fire.

Regardless we are unsubscribing as soon as the component unmounts after logging out.

please let me know if something isn't clear above, and let me get back to you about the share.

stocaaro commented 9 months ago

Hello @mks11 ,

Thanks for the additional input. In my experience useEffect doesn't provide sequencing guarantees. If it's not possible to control when stop and clear call with respect to when observeQuery calls take place, can we catch these errors when they come up as a workaround?

Have you been able to log/follow event sequencing to ensure that they are occurring in the order you expect?

I would really like to see this error happen in my environment, but I'm missing details about how your app renders components. Things I run into trying to repro: use client associated with some but not all components, unclear how import and setup happen server vs client context, missing how UserContext is defined and how it fits into the app. Some of these pieces probably aren't needed for repro, but trying to build an app that works like yours is a challenge without more complete information.

Thanks, Aaron

mks11 commented 9 months ago

Hi @stocaaro,

I just shared a Todo version of the app with you, please let me know if you'd need anything from me (not sure if I need to create a user account for you).

stocaaro commented 9 months ago

Thank you! I'm spinning it up on my own account and will create an account for myself. After logging in, I've done some clicking around and added a button to create todo's to see if that would help me trigger the issue.

Here's a screenshot of what I'm seeing.

I tried logging in and out in quick succession and still haven't triggered the error from above. Do you know what sequence of steps might help me trigger the error?

I really appreciate your work to pull this together!

Regards, Aaron

mks11 commented 9 months ago

Thank you @stocaaro, that's strange because I am seeing it even inside a single tab. But I'd like to add that this happens more consistently when you have another tab open, maybe you could try keeping a tab open (logged in), and login/logout in the current tab inside the same window? Thanks again!

stocaaro commented 9 months ago

That did it. Using multiple tabs, I was able to reproduce the issue you describe.

Reproduction steps:

Create a datastore/auth application that calls clear/stop on logout
Open your application on two tabs, logged in so datastore is running
Log out on one tab, then log back in

Observe that the tab you logged out and then back in on hangs. To un-hang this tab, you can close the other tab and refresh.

This happens because the DataStore.clear() call will only drop the local database (IndexDB) if no other tab is accessing it. Since clear never succeeds, all other datastore processes hang waiting on this to resolve.

This is one of a couple datastore issues happen when multiple tabs are accessing the same local IndexDB instance at the same time.

Better support for multi-tab/window is on our backlog. I can't find another case that better documents this issue with clear/stop, so I'm going to leave this open marked as a feature request.

The issue isn't specifically related to your observeQuery or query calls, having two windows with DataStore started and running means that a clear call in one of them will await until the other window is closed. I've asked around a bit, but don't have a work around recommendation at this time.

cwomack commented 9 months ago

Labeling this as a feature request, as we don't support cross tab signaling out of the box at this point.

aws-amplify / amplify-js