Shim SvelteKit runtime import aliases / Importing `$app/*` fails

[madeleineostoja]

Describe the bug As far as I can tell there's no way to use sveltekit runtime imports (eg: $app/navigation) outside of sveltekit dev/build. Which makes testing virtually impossible. If there is a way to shim these imports outside of the main sveltekit context I haven't found it, so perhaps documentation is needed.

My particular use-case is with Storybook, where UI components that rely on any sveltekit modules break the whole setup. I tried aliasing them with webpack (pointing to .svelte-kit/dev/...) but that didn't work either.

Another use-case is publishing components for sveltekit that would need to rely on those imports.

To Reproduce

1. Setup storybook with Sveltekit
2. Create a component that imports a runtime module (eg: $app/env)
3. Run storybook and see if fail (cannot resolve module $app/env)

Severity Not blocking, but makes building a component library with Storybook or other development/testing frameworks impossible. So, severe annoyance?

[benmccann]

This sounds loosely related to allowing SvelteKit to build components as suggested in https://github.com/sveltejs/kit/issues/518

[madeleineostoja]

Yep that would solve the second use case (distributing components/actions that rely on sveltekit), but I don’t think it would address the first? Testing etc components of a sveltekit app outside the main sveltekit dev/build runtime? Storybook being a very common use case

[alexkornitzer]

TBH I am hitting this issue trying to just do basic testing with uvu and typescript. I have ts files that import $app/env and uvu fails to resolve this using ts-node but I know you guys have set testing as a post 1.0. But any hacks or workarounds would be greatly appreciated. https://github.com/sveltejs/kit/issues/19

[josephspurrier]

Are we able to use something like https://github.com/eirslett/storybook-builder-vite to help with this - or is this a problem outside of Vite?

[josephspurrier]

@madeleineostoja for Storybook, in the .storybook/main.js/cjs, can you try using preprocess.replace like this for each of your aliases (https://github.com/sveltejs/svelte-preprocess#replace-values):

const path = require('path');
const sveltePreprocess = require('svelte-preprocess');

module.exports = {
    stories: ['../src/**/*.stories.mdx', '../src/**/*.stories.@(js|jsx|ts|tsx|svelte)'],
    addons: ['@storybook/addon-links', '@storybook/addon-essentials', '@storybook/addon-svelte-csf'],
    svelteOptions: {
        preprocess: [
            sveltePreprocess({
                replace: [['~', path.resolve('./src')]],
            }),
        ],
    },
};

[josephspurrier]

Ah, I see the challenge - the modules like $app/env are the problem, not just aliases. My workaround won't help there.

[madeleineostoja]

Yeah this isn't to do with your own custom aliases, but the runtime modules of sveltekit. There needs to be a way to consume or shim them in non-sveltekit contexts like testing. Updated issue title to be clearer on the issue

[Dan6erbond]

I'd like to voice my support for a feature that would allow libraries to neatly get access to the $app/navigation, $app/stores, and $app/env modules.

As the creator and maintainer of SvelteKitAuth we're trying to provide a way for users to augment the SvelteKit session with authentication data using the getSession() hook, and upon changes in the session it would be nice to reset it internally instead of expecting users to do something like signOut().then(session.set), and a similar story for routing. Since signIn() either generates a redirect URL or sends a direct fetch() request depending on the payload provided, currently we're returning the URL and expect users to route themselves with goto() as such:

signIn().then(goto);

Letting libraries handle these things internally means less boilerplate for our users, so getting access to the SvelteKit router in a global module, instead of a scoped one would be useful. This is how other frameworks and libraries such as the Vue Router and React Router handle this, as they make use of the global React instance SvelteKit might have to work around the fact that Svelte doesn't provide such a thing, but create its own global context or a singleton.

[madeleineostoja]

I've fixed this in storybook by using their new vite-builder and adding manual aliases to sveltekit's $app runtime module

async viteFinal(config) {
    config.resolve.alias = {
        $app: path.resolve('./.svelte-kit/dev/runtime/app')
    }

    return config;
}

With the caveat being that you have to have run sveltekit dev first to generate those runtime modules. I think this is worth documenting (the path to the alias if nothing else) for others that need to shim these modules until Svelte comes up with an official workaround

[benmccann]

Here's an example of someone having to mock this stuff out for testing: https://github.com/rossyman/svelte-add-jest/issues/14#issuecomment-891387235

[patrickleet]

Author of comment of the said person who had to mock this stuff. I don't mind mocking - fine as a workaround for unit testing anyway. It wasn't exactly simple to figure out the correct mocks though.

[benmccann]

It looks like import.meta.env also might need to be mocked: https://github.com/sveltejs/kit/pull/2210#discussion_r691627049

[wallw-teal]

Here's the test mock we've been using to handle $app/stores. It doesn't handle things like import.meta.env.

<script>
  import {setContext} from 'svelte';
  import {writable} from 'svelte/store';

  export let Component;

  export let stores = {
    page: writable('/'),
    navigating: writable(null),
    session: writable(null),
  };

  setContext('__svelte__', stores);
</script>

<svelte:component this={Component} {...$$restProps} />

So you just pass it the actual component to be tested and (optionally) the stores you want to use for $app/stores.

EDIT: Added spreading of other props onto the component to be tested.

[benmccann]

I think there's a couple ways to do test setup thus far:

• There's the TestHarness.svelte option mentioned above (https://github.com/sveltejs/kit/issues/1485#issuecomment-902965385) along with babel-plugin-transform-vite-meta-env
• Or do mocking. For ESM, Jest has a brand new, experimental, and undocumented unstable_mockModule

I'm not sure what the tradeoffs are and which is the better approach.

Also, if there's anything SvelteKit can do to make testing easier I'd be happy to support changes there.

Related, there's a request to mock fetch (https://github.com/sveltejs/kit/issues/19#issuecomment-914178415), which I haven't seen anyone do yet.

[rmunn]

As the one who made that request to mock fetch, I thought I'd give an update on what I've found so far:

1. When testing components with svelte-jester (via https://github.com/rossyman/svelte-add-jest), I'm outside the Svelte-Kit environment, so load() functions are never run, and I can just pass props to the component to provide it with test data. This reduces the need to provide a mock fetch in unit testing scenarios.
2. When doing E2E tests with Playwright, I can use Playwright's page.route feature to intercept browser requests to specific URLs and return my test data instead of what that URL would have returned (and the URL is never hit). This works only in the browser that Playwright is driving, and does not work server-side. But I can add kit: { ssr: !(process.env.NODE_ENV === 'test') } to my svelte.config.js to turn off server-side rendering, so that load() is only ever run on the client where Playwright can intercept it.
3. If I want to run E2E tests in an environment as close to production as possible, though, I'd want to run them with server-side rendering turned on. I could, in theory, use externalFetch to rewrite URLs from loaded pages to return test data instead. But that doesn't help me with internal URLs like /api/blogpost/3, where externalFetch is not called.

So without the ability to mock fetch, I can run almost all the test scenarios I need. I can unit-test my components, and I can run E2E tests with SSR turned off and intercept the fetch requests in the E2E browser. The only thing I can't do is run E2E tests with server-side rendering turned on. For that scenario, I believe I would need to be able to mock the fetch function passed into load().

[patrickleet]

@rmunn

so load() functions are never run

In at least pre-esm jest@26 and svelte-jester@1, you can mock $app/env.js to set browser to true or false, which will cause the svelte component to server side render or client side render - the load function is called when ssr'd.

Theoretically this should now also be possible in ESM jest@27 and svelte-jester@2, via the new unstable_mockModule mocking method, but I have not tried that yet.

jest.mock('$app/env.js', () => ({
  amp: false,
  browser: false,
  dev: true,
  mode: 'test'
}))

In my tests directory, I find it simplest to have two test files index-client.js and index-server.js, with the browser set to true and false, respectively.

Here's an image of a coverage report showing lines of the load function being hit:

Maybe I'm missing something about mocking fetch, but you should be able to just mock fetch in the same way. jest.mock('fetch', ...)?

In the same code this is pulled from, my load function is loading graphql over HTTP, however I am using a graphql library for that, not fetch directly, and thus, just mock the graphql library to return the responses I would get in good, bad, and other important cases. Similarly, if you were using axios instead of fetch, you could mock the response of the .get or whatever. You don't want to rely on networks in unit tests as that is outside of the scope of the unit. Things like that are best left for integration tests.

[patrickleet]

Other sveltekit mocking tips:

import.meta.env

1. Put all import.meta.env usage in one file, I have been naming that file lib/env.js, so I can reference it via $lib/env.js. This way in the majority of situations you can just mock that one file to set the envs you want to use for the test context:

lib/env.js

export const VITE_HASURA_GRAPHQL_URL = import.meta.env.VITE_HASURA_GRAPHQL_URL
export const VITE_HASURA_GRAPHQL_WS_URL = import.meta.env.VITE_HASURA_GRAPHQL_WS_URL

test.js

// ENV mocks
jest.mock('$lib/env', () => ({
  VITE_HASURA_GRAPHQL_URL:
    'http://fakeendpoint.example.com/v1/graphql',
  VITE_HASURA_GRAPHQL_WS_URL:
    'ws://fakeendpoint.example.com/v1/graphql'
}))

$app/navigation.js

jest.mock('$app/navigation.js', () => ({
  goto: jest.fn()
}))

[rmunn]

@patrickleet wrote:

Here's an image of a coverage report showing lines of the load function being hit:

What's the setup you're using for those tests? I've looked at https://github.com/CloudNativeEntrepreneur/sveltekit-eventsourced-funnel, but there you're using render() from @testing-library/svelte, which is the same thing that I'm using that doesn't call load(). (That is, it doesn't call the load() function from the module context the way Svelte-Kit does; it's just using Svelte to compile the component). I haven't yet found an example of running Jest tests in a Svelte-Kit context. Your coverage report shows that that's what you're doing, so I'd be interested to see how you've set that up. Would you be able to share your Jest config and/or a couple of your unit tests that are running Svelte-Kit's load() function, so I have an example of how to do that?

[patrickleet]

@rmunn

This is from a different repo for a client, so it's private, unfortunately - it's still on jest@26 / svelte-jester@1, and the "funnel" example was more of an example of event sourcing with svelte than a unit test demonstration - just happened to be public and relevant to some of the recent testing changes.

That said - let me see about pulling out relevant pieces...

Ok - looked ... Looks like I just exported/imported load in the server test.

companies/index-server.js

/**
 * @jest-environment jsdom
 */
import '@testing-library/jest-dom/extend-expect'
import { render } from '@testing-library/svelte'
import companiesIndex, { load } from '$routes/companies/index.svelte'
import debug from 'debug'

const log = debug('tests')

log('starting suite routes/companies/index.svelte')

// Sveltekit Mocks
jest.mock('$app/env.js', () => ({
  amp: false,
  browser: false,
  dev: true,
  mode: 'test'
}))

jest.mock('$app/navigation.js', () => ({
  goto: jest.fn()
}))

// In more recent tests I've started using the "TestHarness" instead of this `svelte` mock with a fake getContext
jest.mock('svelte', () => {
  const { writable } = require('svelte/store')
  const actualSvelte = jest.requireActual('svelte')
  const fakeGetContext = jest.fn((name) => {
    if (name === '__svelte__') {
      return fakeSvelteKitContext
    }
  })
  const fakeSvelteKitContext = {
    page: writable({
      path: '/',
      query: new URLSearchParams({
        offset: 0,
        limit: 5
      })
    }),
    navigating: writable(false)
  }

  const mockedSvelteKit = {
    ...actualSvelte,
    getContext: fakeGetContext
  }
  return mockedSvelteKit
})
// End Sveltekit mocks

// ENV mocks
jest.mock('$lib/env', () => ({
  VITE_PRESIDIO_HASURA_GRAPHQL_URL:
    'http://fakeendpoint.example.com/v1/graphql',
  VITE_PRESIDIO_HASURA_GRAPHQL_INTERNAL_URL:
    'http://fakeendpoint.example.com/v1/graphql',
  VITE_PRESIDIO_HASURA_GRAPHQL_WS_URL:
    'ws://fakeendpoint.example.com/v1/graphql'
}))

// Network mocks
jest.mock('$lib/data/urql', () => ({
  client: {
    query: jest.fn(() => {
      const result = {
        data: {
          companies: [
            {
              id: 'test-1',
              name: 'Test 1',
              logo_url:
                'https://res.cloudinary.com/crunchbase-production/image/upload/v1418896144/nzn3gfio6p8lupehf6nv.jpg',
              __typename: 'companies'
            },
            {
              id: 'test-2',
              name: 'Test 2',
              logo_url:
                'https://res.cloudinary.com/crunchbase-production/image/upload/v1397199104/34852c1debc24e028c4082caa0efb427.jpg',
              __typename: 'companies'
            },
            {
              id: 'test-3',
              name: 'Test 3',
              logo_url:
                'https://res.cloudinary.com/crunchbase-production/image/upload/rswshbdwsa7bg39kjtjk',
              __typename: 'companies'
            }
          ],
          companies_aggregate: {
            aggregate: {
              count: 3
            }
          }
        }
      }

      return {
        toPromise: jest.fn(() => Promise.resolve(result))
      }
    })
  }
}))

// mock store that uses URL query string
jest.mock('$lib/stores/queryStore')

const ctx = {
  page: {
    query: {
      get: jest.fn((key) => {
        const params = {
          limit: 50,
          isClientFilter: true,
          order: 'asc',
          offset: 0
        }

        return params[key]
      })
    }
  }
}

describe('routes/companiesIndex.svelte - server', () => {
  // browser false is default mock

  describe('server side rendering', () => {
    it('should server render empty', async () => {
      const { getByText } = render(companiesIndex)
      expect(getByText('Companies')).toBeInTheDocument()
      expect(getByText('No companies')).toBeInTheDocument()
    })

    it('should server render empty with data but empty companies result', async () => {
      const props = await load(ctx)
      props.companies = []
      const { getByText } = render(companiesIndex, { props })
      expect(getByText('Companies')).toBeInTheDocument()
      expect(getByText('No companies')).toBeInTheDocument()
    })

    it('should server render with data', async () => {
      const { getByText } = render(companiesIndex, await load(ctx))
      expect(getByText('Companies')).toBeInTheDocument()
      expect(getByText('Test 1')).toBeInTheDocument()
    })
  })

  describe('#load', () => {
    it('should query graphql endpoint and return found companies', async () => {
      const { client } = require('$lib/data/urql')
      let result = await load(ctx)
      expect(client.query).toBeCalled()
      expect(result.props.companies.length).toBe(3)
    })
  })
})

And the load function from that component:

  export async function load({ page }) {
    const variables = {
      limit: parseInt(page.query.get('limit'), 10) || defaults.limit,
      isClientFilter: page.query.get('isClientFilter') !== 'false',
      nameFilter: `%${page.query.get('nameFilter') || ''}%`,
      order: page.query.get('order') || 'asc',
      offset: parseInt(page.query.get('offset'), 10) || defaults.offset
    }

    const result = await client.query(QUERY, variables).toPromise()
    const { data } = result
    const { companies } = data

    return {
      props: {
        companies,
        count: data.companies_aggregate.aggregate.count
      }
    }
  }

.babelrc

{
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": "current"
        }
      }
    ]
  ]
}

jest.json

{
  "roots": ["<rootDir>/src", "<rootDir>/__tests__/unit"],
  "testEnvironment": "node",
  "modulePaths": ["<rootDir>/src"],
  "moduleDirectories": ["node_modules"],
  "transform": {
    "^.+\\.svelte$": "svelte-jester",
    "^.+\\.(ts|tsx|js|jsx)$": ["esbuild-jest"]
  },
  "moduleFileExtensions": ["js", "svelte"],
  "moduleNameMapper": {
    "^\\$app(.*)$": "<rootDir>/.svelte-kit/build/runtime/app$1",
    "^\\$lib(.*)$": "<rootDir>/src/lib$1",
    "^\\$routes(.*)$": "<rootDir>/src/routes$1"
  },
  "setupFilesAfterEnv": ["@testing-library/jest-dom/extend-expect"],
  "coverageThreshold": {
    "global": {
      "branches": 0,
      "functions": 0,
      "lines": 0,
      "statements": 0
    }
  },
  "collectCoverageFrom": ["src/**/*.{js,svelte}"],
  "testTimeout": 30000
}

[brev]

If you don't mind using experimental features of the latest Node.js, along with esmodules/import/export, you can use an esmodule loader hook to mock SvelteKit's $app/navigation-style import aliases.

Rough beginner example link follows. One could mock the module into a no-op, or rewrite it to .svelte-kit/dev/runtime/app for real functionality (after svelte-kit dev has run):

• https://www.npmjs.com/package/create-esm-loader#2-create-directory-aliases

I've built a loader for .svelte files, which can also do pre-processing, and no-op imports of .css, pre-processed assets, and now SvelteKit's import { goto } from '$app/navigation':

• https://github.com/brev/esm-loader-svelte

This allows me to simply test my .svelte components in node/es6/esm with uvu, like @alexkornitzer but without typescript.

• Context: https://github.com/lukeed/uvu/issues/122

These loader hooks do not chain well yet, so it's far from a perfect solution, in general.

EDIT: Big update in my next comment below!

[brev]

As an update to my previous post, I've made a lot of new progress, details are here:

https://github.com/sveltejs/kit/issues/19#issuecomment-1041134457

Now supporting:

• Handling import.meta.env, etc.
• Handling aliases like $lib and $app/*.
• Mocking in general (fetch, etc.) Also, mocking of Stores before component rendering to provide context to components tested in isolation.

[Rich-Harris]

Copy-pastable repro here: https://github.com/sveltejs/kit/issues/4432

[benmccann]

https://github.com/michaelwooley/storybook-experimental-vite demonstrates setting up Storybook with the new vite.config.js. We still need to figure out how to handle $navigation and $stores

[benmccann]

There's a great example of Vitest mocks here: https://github.com/sveltejs/kit/issues/5525#issuecomment-1186390654

[prakhar-pal]

I'm facing similar issue in a sveltekit app. The error message is Uncaught TypeError: Cannot read properties of undefined (reading 'disable_scroll_handling') at navigation-15d5f540.js: which causes 500 status error. I've read through issue https://github.com/sveltejs/kit/issues/4432 as well and it doesn't seem applicable. The issue happens randomly but frequently in production mode and in low speed networks.

Any idea?

[peterpeterparker]

Got this issue as well when I use afterNavigate in a +page while using the adapter static and chunking strategy of vite (splitVendorChunkPlugin).

[peterpeterparker]

Sample repo btw. https://github.com/peterpeterparker/my-app-yolo

[benmccann]

@peterpeterparker your issue sounds different than what this one is talking about. Can you file a new issue for it?

[peterpeterparker]

Sure. I just commented here because #4432 was closed and linked with the issue.

So here you go 👉 https://github.com/sveltejs/kit/issues/7415

[vhscom]

The introduction of $env has made the surface of this issue widen from just $app/*. Without the ability to shim these things unit/component testing becomes increasingly difficult (ref: https://github.com/microsoft/playwright/issues/18465) and, in practice, leads to awkward workarounds in order to properly support testing environments such as Playwright component testing.

[benmccann]

Most aliases are now supported out-of-the-box with the latest Storybook 7. You can see a summary of what is supported and not supported here: https://github.com/storybookjs/storybook/tree/next/code/frameworks/sveltekit

There are a few that are not yet supported and make more sense to support as mocks. I've created a new issue to track that: https://github.com/storybookjs/storybook/issues/20999. PRs for it would be very welcome!

[madeleineostoja]

So good! I originally opened this for the storybook use-case, and that seems like it's met now (or could be closed out in favour of the new issue you opened).

I guess the only thing left for this issue is shimming them in other context, like test suites etc.

[benjaminpreiss]

Hey all! I have been searching for the correct issue, but all other issues about importing $app/stores in a library build are sadly closed and have not lead to a solution.

Currently, I am getting the error Cannot find package '$app' imported from /Users/benjaminpreiss/Documents/work/montee/node_modules/@frontline-hq/sveltekit-i18n/index.js for this line of code in my sveltekit library: https://github.com/frontline-hq/sveltekit-i18n/blob/09b9c7b62e17661a7d5c8dbf1585132ff7b29f0a/src/lib/index.ts#L2

Does anybody know how to enable $app/stores imports in library build specifically for sveltekit?

[madeleineostoja]

Now that we're post v1.0 do we have a better testing story for sveltekit modules? Looks like the .svelte-kit/runtime alias hack is no longer viable, and if we're going to mock all of these modules (which cover quite a large api surface now) it would be great it sveltekit provided the mocks, either within @sveltejs/kit in an an 'official' testing utils package.

It's kinda crazy that component unit testing doesn't work OOTB yet with a v1 framework

[ivanhofer]

@benjaminpreiss and anyone using $app in a package released to npm: The application that uses your library has to add the following to their vite.config.js

ssr: {
    noExternal: ['your_package_name'],
}

[iamthe-Wraith]

Any update on this? I really would love to be able to use Playwright component testing with Sveltekit, but this issue is sadly preventing it and keeping us from being able to adopt Svelte(Kit) on a larger scale...

[dskloetd]

I would also like to use Playwright component testing with SvelteKit. Does anyone know if there is a work-around that works for the case of using Playwright component testing?

[amosjyng]

For anyone else that gets the error

Error: Cannot subscribe to 'page' store on the server outside of a Svelte component, as it is bound to the current request via component context. This prevents state from leaking between users.For more information, see https://kit.svelte.dev/docs/state-management#avoid-shared-state-on-the-server
 ❯ get_store node_modules/@sveltejs/kit/src/runtime/app/stores.js:96:9
 ❯ Object.subscribe node_modules/@sveltejs/kit/src/runtime/app/stores.js:39:37                                                                      
 ❯ subscribe node_modules/svelte/src/runtime/internal/utils.js:139:22
 ❯ Module.component_subscribe node_modules/svelte/src/runtime/internal/utils.js:159:31                                                              
 ❯ instance src/Demo.svelte:69:24

I believe that is related to this issue. I had a minimal repro set up for a new issue before realizing it's probably the same as this issue. Run

$ yarn
$ yarn test run

to reproduce. On commit 388142a, it works. After this commit, the test breaks in the manner mentioned above.

[opack]

We found some solution to work around not being able to mock those modules. In case it can help someone in the same situation, and in order to gather some feedback about what we found, I write below the whole story: our context and our solution.

Some context

Just to explain how this “shimming” this is crucial to us, and how we really tried to avoid is… in vain :-(

For our project, we are using TDD, SvelteKit and Superforms. So, be able to test is crucial for us. That being said, we ran into this “mocking” problem before with $env and resolved it easily, but when trying to implement Superforms we had a real blocker.

We spent a lot of time on the Internet, browsing issues, testing solutions. But as there seem to be no way to correctly shim the $app modules (especially $app/forms and $app/navigation, which both contain some complex logic methods such as beforeNavigate, enhance or applyAction), we had to choose between these solutions:

• Abandon Superforms: we could try to mock some $app modules, but this would fail when using libraries that also use these modules. So, a solution would be to abandon thoses librairies. Problem is that there is not always a good alternative not using thoses Svelte modules, and it is most certain that at some point we run into another library using thoses modules.
• Abandon TDD: no way! We are a 2-person team, so we cannot afford to take the risk to not have the safety net and peace of mind TDD gives us. Plus the other benefits, of course, but these two are important for us.
• Abandon SvelteKit: we choose SK over React when we started, but it is still an option, although we do not want because we love this framework so much!

Our workaround solution

As no solution was valid, we tried to change a bit our way of doing things. Currently, we do component testing with Playwright experimental component testing feature. But in the end, component testing is just end-to-end testing with a specific page containing only one component. This seems to be indeed how SvelteKit seems to be tested: small “apps” exposing only a specific feature to be tested.

So we implemented this:

• Have a route for each component to test, leading to a page which would only contain this component. We put those in src/routes/tests and have some conventions for the folders beneath:
  • Mimic the folder structure of the component to test. For example, when testing a src/routes/(auth)/signup/+page.svelte, we need to create a src/routes/tests/(auth)/signup/+page.svelte. When testing a src/lib/components/form-input/form-input.svelte, we need to create a src/routes/tests/lib/components/form-input/+page.svelte.
  • If multiple “forms” of the component have to be tested (same as using multiple mount in a Playwright component test), then we create a subfolder with a _ prefix and a name explaining how this feature is different. For example, if we need to test a the FormInput component mentioned above in a situation that it would receive a value parameter, then we create src/routes/tests/lib/components/form-input/_has-initial-value/+page.svelte.
• Have a Playwright test for each of these routes, same way we would do for our components. We colocate these tests with our components, as before, but they can be put anywhere in src.
• Have a Playwright configuration dedicated to this component testing that targets only these tests.
• This solution will need to have a dev server running (with npm run dev for example) so that we can run our tests and most importantly benefit from HMR.

Reading test results in the Playwright test

To be able to make this work, we also had to find a way to retrieve some result value from the tested component. For example, how to test that a button component correctly calls a callback when it is submitted? In e2e testing, this can be tricky because we only browse as a user do, we do not have access to the code. For this purpose, we created a utility component and method to help displaying and retrieving data:

// test-result.svelte
<script lang="ts">
    export let id = 'test-result'
    export let value: unknown
</script>

<span data-testid={id} class="hidden">{JSON.stringify(value)}</span>

// test-result.util.ts
import type { Page } from 'playwright-core'

export const getTestResult = async <T>(page: Page, id: string = 'test-result'): Promise<T> => {
    const resultText = await page.getByTestId(id).innerText()
    return JSON.parse(resultText) as T
}

We then use it like this:

// src/routes/tests/lib/components/form-input/_has-initial-value/+page.svelte
<script lang="ts">
    import FormInput from '$lib/components/form-input/form-input.svelte'
    import TestResult from '$routes/tests/test-result.svelte'
    import { generateRandomUsername } from '$tests/unit/utils/string-generator.util'

    const initialUsername = generateRandomUsername()
</script>

<FormInput value={initialUsername} />
<TestResult value={initialUsername} />

// $lib/components/form-input/form-input.spec.ts
test('has the passed value by default', async ({ page, pom: { username } }) => {
        await expect(username).toHaveValue(await getTestResult(page))
})

What’s cool

We are just begining to use this, so maybe I’ll come back in a few days/weeks and tell it’s totally lame to do that 😅 But what we see in this solution are the following perks:

• It’s really close to what we would do with Playwright component testing, except that Playwright is not dynamically mounting our component in a template page.
• We use the component in a way that is even closer to its real usage: we create a real page with real code similar to how our component would be used, passing attributes the same way, using callbacks and bindings the same way etc.
• We use the real runtime:
  • no need to mock anything in SvelteKit, so everything works, and of course Superforms does also.
  • we can use and therefore test everything that is proposed by SvelteKit, and also things that could not be tested correctly in Playwright component testing (such as <svelte:head> to change the title of the page, or goto)
  • the correct layout is used, so we can really benefit (and test) from anything done at this level
  • it’s possible to see and interact with what’s really rendered, if we want, by just browsing to the URL (because we have a dev server running, which is used by our tests)
• HMR: as we have a dev server running background, we benefit from HMR, which is faster that the building/server executed by Playwright component testing when we run a test from VSCode
• We do not need to do this little hack in some component tests to mount the component even if we do not use it, just to make sure the component is rendered (for example when running accessibility tests).

What’s not so great

There are some downsides to this technique:

• As explained before, if the component has to be tested in different configurations (which would correspond to a new call to mount in Playwright component testing), then we need to create 1 route (which means 1 subfolder and 1 +page.svelte) for each of these configurations. Fortunately, we should not have a lot of different configurations for our components.

• The bigger one: we need to find a way to remove testing routes from the final bundle. Or course, we do not want to ship those tests routes, so we want to tell Vite to just ignore what is in src/routes/tests. Unfortunately, it does not seem to be possible. I spent a good amount of time on the Internet, tried some things (vite configuration, Rollup configuration, creating a vite plugin) but nothing worked. The only solution that I found is to use renamer to change the name of the files before and after build. Our NPM script becomes:

  // package.json
  ...
  "build": "renamer --find \"+\" --replace \"__\" \"src/routes/tests/**\" && vite build && renamer --find \"__\" --replace \"+\" \"src/routes/tests/**\"",
  ...

  The caveat with this being that some strange thing could occur if you edit or rename the file elsewhere during this process.

  I would love to have a simpler way to do this (just like “hey vite, completely ignore this folder please”), so if anyone has an idea, please send me a message 😬

What I would love from SvelteKit

I hope this will help someone, but I would love to have some things from SK out-of-the-box:

• An official and neat way to mock thoses runtime modules
• A way to strip some routes from the bundle (this can also be used to strip some admin or dev-only routes, as I saw in some discussions)

Please feel free to tell me what you think about this solution, and if I missed something. We will continue on this path and I will come back here and edit if we found something interesting, especially if everything turned out to be garbage in the end 😅 (but I hope not 😉)

[opack]

Hi! As promised, here's some feedback after using this technique for 2 weeks. It's working rather well regarding the initial problem of not having a way to mock $app modules. However, we are having some issues:

• 👎 having test routes is not ideal because it's not easy to strip them from the bundle, and they cannot be near the components they tests, which make it hard to maintain tests
• 👎 tests are rather slow and Playwright as a test runner lacks the ability to rerun a test when corresponding code has changed, which is even more of a problem with this technique, so it's not really fluid to develop like that
• ⛔ last but not least: we are having lots of flaky tests with Playwright these days, and it's getting hard to continue. We are considering leaving Playwright and going back to Vitest (+ Testing Library). However, we would still have this $app modules problem with nos solution (as Vitest has no mean to go to a URL to perform tests, nor can it launch a headless browser as Playwright does).

@benmccann I know you're all very busy and that there's a lot of work to be done on this framework that we all love. Nevertheless, in the absence of any feedback from the team on this subject, I can't help but wonder how this work is being prioritized. Indeed, it is (in my opinion) crucial to the widespread adoption of SvelteKit, given that if it's impossible to properly test our applications (which is unfortunately the case today) then SvelteKit (and therefore Svelte) will probably not be chosen in many contexts.

In any case, that's the conclusion we've reached. Today, we'd like to have some feedback on the implementation of a solution to this problem, as well as a possible delivery target. I'm well aware that the notion of a deadline on such a project is a very complicated one, but we need to know whether we can expect to wait a few weeks, or whether we're talking months or years, in which case we'll have to turn to another framework.

I'm not sure I'll be able to help more directly by contributing code, but if possible please feel free to give some general lines on what needs to be done, in case I or someone passing by can lend a hand on the subject 😉

Thanks again for your work, really, and I hope this topic can move forward! 🙏

[benmccann]

Storybook now has built-in mocks for SvelteKit: https://github.com/storybookjs/storybook/pull/24795

[opack]

Hi! I also received the notification for this, but it does not help for using Playwright Component Testing 😩