Filters are not working on deployed site

archiewood commented 3 months ago

Steps To Reproduce

Removed .evidence folder
Removed .build folder
Ran npm run sources
Ran npm run build
cd build
Host folder using http: http
Open http://localhost:8000/ in new private windows (Firefox Developer Edition)

Environment

"dependencies": {
    "@evidence-dev/bigquery": "^2.0.3",
    "@evidence-dev/core-components": "^3.2.0",
    "@evidence-dev/csv": "^1.0.4",
    "@evidence-dev/databricks": "^1.0.2",
    "@evidence-dev/duckdb": "^1.0.4",
    "@evidence-dev/evidence": "^26.0.0",
    "@evidence-dev/mssql": "^1.0.4",
    "@evidence-dev/mysql": "^1.0.2",
    "@evidence-dev/postgres": "^1.0.2",
    "@evidence-dev/snowflake": "^1.0.2",
    "@evidence-dev/sqlite": "^2.0.2",
    "@evidence-dev/trino": "^1.0.3",
    "mermaid": "^10.7.0"
  },
  "overrides": {
    "jsonwebtoken": "9.0.0",
    "trim@<0.0.3": ">0.0.3",
    "sqlite3": "5.1.5",
    "mysql2": "3.8.0"
  }

Browser: Firefox

Expected Behavior

Actual Behaviour

most things looks and acts normal. But on one page we use dropdowns to filter data. That page stops working (either no change when changing filter or no data at all).

When running npm run dev, directly in same folder this works. I can use the dropdowns and the components referring the data is updated.

andrejohansson commented 3 months ago

Is there a confirmation on this and/or some potential workaround (like: knowing what version to downgrade to?)

mcrascal commented 3 months ago

@csjh can you take a look at this one this week?

archiewood commented 3 months ago

It sounds like the underlying issue is not the all-queries.json specifically, but rather the "too small to be Parquet" error

andrejohansson commented 3 months ago

@archiewood I doubt it, I have this problem on pages that do not have the too small to be parquet error also. But the all queries is there. Root cause seems to be some misalignent with the hashes (data is there, just in other folders with other hash values).

archiewood commented 3 months ago

@andrejohansson do you have or could you make a repro project that reproduces this error that you can share

We are struggling to reproduce this consistently internally!

andrejohansson commented 2 months ago

It seems that the 404 all-queries errors come whenever there is a markdown page which have no queries (I guess it makes sense). So this yields a missing all-queries and the 404 error. But this does not seem to relate to the dropdowns not working, will continue my efforts in locating the issue.

andrejohansson commented 2 months ago

Here is a sample that reproduces the behaviour (no update on filter change).

evidence-issue-1566.zip

Preparation

npm install
npm run sources

Working Scenario

Start dev server
```
npm run dev
```
Open browser http://localhost:3000/
Use the filter
Confirm bar chart updates

Not Working Scenario

Build static page
```
npm run build
```
Start some web server hosting static pages, I am using https://github.com/cortesi/devd
```
cd build
devd .
```
Open browser http://localhost:8000/
Use the filter

Confirm bar chart does not update anymore

Console now contains error:

Uncaught (in promise) Error: Invalid Input Error: File 'csv_data_ingestion_system_type_statistics.parquet' too small to be a Parquet file

Observations

My folder name is called csv and the source name in connection.yaml is called csv_data
- Do these need to be in sync?
  - Renaming the folder to csv_data did not help
My problem seems to come from the data itself, especially how dates are coming from csv to duckdb (the date column have dates in YYYY-MM-DD format)
Trying to convert the csv data to a sqlite file also gave me some trouble
- Sqlite Documentation is unclear/wrong
  
  SQLite is a local file-based database. It should be stored in the root of your Evidence project.
  
  This does not seem to be true. I must place the sqlite file under sources\<mydatasourcename> and also add a connection.yaml file similar to
```
name: agentdata
type: sqlite
options:
  filename: agentdata.sqlite
```
  This works to get the data, but generates a warning when I run npm run sources as it tries to process all files in the source folder (including the sqlite file itself) and not only the .sql files

andrejohansson commented 2 months ago

In order to exclude any date issues, i created a duckdb datasource instead of csv of the same data.

Here is a variant using duckdb:

evidence-issue-1566-using-duckdb.zip

But the problem persists, there is no interactivity in when filtering using the dropdown after being built with npm run build. Filtering only works using the dev server.

Sidenote: I am using windows 11

archiewood commented 2 months ago

Thank you for the detailed repro

My folder name is called csv and the source name in connection.yaml is called csv_data. Do these need to be in sync?

No - your setup is fine

Sqlite Documentation is unclear/wrong

Thank you, I have opened a PR to fix this

Confirm bar chart does not update anymore

With your csv example, I am unable to replicate this issue on my local machine.

I have also deployed this repository on netlify, and the filter appears to work. Does this also work for you on windows?

https://magical-baklava-cef8ed.netlify.app/

CleanShot 2024-02-20 at 09 52 27

andrejohansson commented 2 months ago

Yes, the netlify app works for me too
Do you host using some static webserver and not the dev?

Here is a screencast of my behaviour, interestingly enough I can seem to return back to the "all" filter.

https://github.com/evidence-dev/evidence/assets/2828428/64674699-eef7-40b0-b037-a60e21ac6485

archiewood commented 2 months ago

On local i tried both the built in vite preview npm run preview and http-server.

My best current guess on this is a platform specific issue to windows.

What is your current hosting setup where this is failing?

andrejohansson commented 2 months ago

Like I wrote above, I just npm run build to the build folder then start devd in it (something similar to http-server).

I noticed that when I use the filters, only very small datasets are returned (7 rows each). But when I use the % operator to show all, then I get 1k+ rows and then the charts and everything work.

Could there be some limit that is wrong and prevents really small datasets?

Uncaught (in promise) Error: Invalid Input Error: File 'duckdata_measurements.parquet' too small to be a Parquet file
    Immutable 3
[VennDiagram.svelte_svelte_type_style_lang.96597554.js:4:56898](http://localhost:8000/_app/immutable/chunks/VennDiagram.svelte_svelte_type_style_lang.96597554.js)

archiewood commented 2 months ago

Okay, I have now downloaded devd and replicated the issue.

CleanShot 2024-02-20 at 10 21 58@2x

While I don't want to exclude the possibility of errors on our side - which we'll investigate, perhaps as a workaround using a post 1.0 webserver will resolve your issue?

andrejohansson commented 2 months ago

npm run preview (the vite preview) works well for me too.

I confirmed also that using caddy with the following command work, so Indeed it is some server configuration

caddy file-server --listen :8000

In our production docker image we are using nginx unit which also shows the same behaviour and this is a brand new webserver.

Do you have any idea what settings my be required or could affect this? I'm thinking something like mime types, or allowing/disallowing something.

andrejohansson commented 2 months ago

The devd webserver responds with octet/stream (which seems right) but is not working

The caddy server responds with xml (which seems wrong) but is working

archiewood commented 2 months ago

@ItsMeBrianD might have some better ideas here, as he was recently deploying evidence using nginx.

I believe there are certain file types that are not served correctly by nginx by default

ItsMeBrianD commented 2 months ago

Do you have any idea what settings my be required or could affect this?

I've not heard of nginx unit before, but it looks very interesting. The setup I was running with was with a ootb nginx docker container. I did need to tweak the file permissions of the .parquet files (they were getting spit out 0700 owned by root)

this is the full configuration that I used:

server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    gzip on;
    gzip_types text/plain application/xml application/octet-stream application/json application/javascript;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location ~* \.(parquet|arrow|js|json)$ {
      root   /usr/share/nginx/html;
      add_header Cache-Control "private, max-age=3600";
   }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

Something that we encountered in another spot was this error appearing if duckdb wasm runs out of memory, is this happening on pages that have a lot of data (or after navigating around the site for a bit?)

andrejohansson commented 2 months ago

Thank you both for your efforts.

No this sample is very small, 7 rows of data when filtered. 1k rows when unfiltered.

Switching webserver to caddy helped locally, I'll see if I can get either caddy into docker or use your sample nginx config @ItsMeBrianD

ItsMeBrianD commented 2 months ago

There is an official caddy docker image if that's helpful

andrejohansson commented 2 months ago

I can confirm to you that the issue was resolved for us when switching to caddy as webserver instead of using nginx and nginx unit.

Here is our simple docker:

FROM caddy:2.7.6

# Copy web server config
# NOTE: Use a compatible web server configuration, see https://github.com/evidence-dev/evidence/issues/1566 
COPY ./Caddyfile /etc/caddy/Caddyfile

# Copy app from build stage into default www folder
COPY --from=builder /app/build /var/www/html

EXPOSE 80

archiewood commented 2 months ago

Excellent, closing for now

csjh commented 2 months ago

Appears to have been a duckdb-based error, as upgrading to duckdb-wasm 1.28.0 fixes, so reopening pending that version bump.

evidence-dev / evidence