search

MDeLuise / plant-it

🪴 Self-hosted, open source gardening companion app
https://plant-it.org
GNU General Public License v3.0
691 stars 26 forks source link

MalformedJsonException when searching in Trefle db #229

Closed astappiev closed 5 months ago

astappiev commented 5 months ago

Avoid duplicated bug reports

Description

After running the server I got the following error when I attempted to search for new plants (using Trefle)

{
    "statusCode": 500,
    "errorCode": "INTERNAL_SERVER_ERROR",
    "message": "com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 12 path $"
}

This error is found in browser logs on /api/botanical-info request. There is nothing that happens on the frontend. Also, the server logs are clear, and nothing suspicious.

Of course, the configuration already includes ignoring certificates of Trefle.

Local environment

Last server logs:

server-1  | 2024-06-29T20:38:11.719Z DEBUG 21 --- [nio-8080-exec-4] c.g.m.p.p.PlantInfoExtractorFacade       : Extract botanical info matching ros (size 5)
server-1  | 2024-06-29T20:38:11.720Z DEBUG 21 --- [nio-8080-exec-4] c.g.m.p.b.BotanicalInfoService           : Search for DB saved botanical info matching 'ros' scientific name (max size 5)
server-1  | 2024-06-29T20:38:11.731Z DEBUG 21 --- [nio-8080-exec-4] c.g.m.p.p.trafle.TrefleRequestMaker      : Fetching info for "ros" from Trefle

Docker compose

name: plant-it
services:
  server:
    image: msdeluise/plant-it-server:latest
    env_file: server.env
    depends_on:
 #     - db
      - cache
    restart: unless-stopped
    volumes:
      - "./upload-dir:/upload-dir"
      - "./certs:/certificates"
    ports:
      - "8042:8080"
      - "8041:3000"
    extra_hosts:
      - "db:host-gateway"
#  db:
#    image: mysql:8.0
#    restart: always
#    env_file: server.env
#    volumes:
#      - "./db:/var/lib/mysql"

  cache:
    image: redis:latest
    restart: always
MDeLuise commented 5 months ago

Hi @astappiev, thank you for opening the issue!

Unfortunately, the Trefle service is no longer operational. The project is inactive, and while the dataset was accessible until now, it is no longer available. I am currently working on a replacement for this service, as mentioned in issue #218.

I will keep everyone updated on the progress. Thank you for your patience and understanding.

astappiev commented 5 months ago

Thank you for replying. Yes, I read that, but the latest update I found is that we need to skip certificate verification.

Do they have their database available? If so, we can deploy it locally perhaps.

MDeLuise commented 5 months ago

but the latest update I found is that we need to skip certificate verification.

It was that way, but unfortunately now Trefle backend has shut down completely, so that fix does not work anymore.

Do they have their database available? If so, we can deploy it locally perhaps.

I tried to build Trefle locally using the Dockerfile provided in their repository. However, I encountered the following error when running:

docker build --no-cache -t trefle-fork -f Dockerfile . --progress=plain

The error output is:

#15 [10/12] RUN mkdir -p /app/tmp/pids &&   chmod -R 777 /app/tmp &&   chmod 777 /app/bin/post-start
#15 DONE 0.1s

#16 [11/12] RUN bundle exec rails assets:precompile --trace --verbose
#16 1.404 rails aborted!
#16 1.406 LoadError: cannot load such file -- nokogiri
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/loofah-2.19.0/lib/loofah.rb:4:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rails-html-sanitizer-1.4.3/lib/rails-html-sanitizer.rb:2:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/actionview-7.0.4/lib/action_view/helpers/sanitize_helper.rb:3:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/actionview-7.0.4/lib/action_view/helpers.rb:8:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/context.rb:1:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/sprockets-rails-3.4.2/lib/sprockets/railtie.rb:10:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails.rb:3:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:73:in `rescue in block in require'
#16 1.406 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:51:in `block in require'
#16 1.406 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:44:in `each'
#16 1.406 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:44:in `require'
#16 1.406 /usr/local/lib/ruby/3.1.0/bundler.rb:176:in `require'
#16 1.406 /app/config/application.rb:19:in `<main>'
#16 1.406 /app/Rakefile:4:in `require_relative'
#16 1.406 /app/Rakefile:4:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:29:in `load'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:29:in `load_rakefile'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:710:in `raw_load_rakefile'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:104:in `block in load_rakefile'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:103:in `load_rakefile'
#16 1.406 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
#16 1.406 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
#16 1.406 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:18:in `perform'
#16 1.406 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/command.rb:51:in `invoke'
#16 1.406 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.406 bin/rails:4:in `<main>'
#16 1.406
#16 1.406 Caused by:
#16 1.416 LoadError: cannot load such file -- sprockets-rails
#16 1.416 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:60:in `block (2 levels) in require'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:55:in `each'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:55:in `block in require'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:44:in `each'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler/runtime.rb:44:in `require'
#16 1.416 /usr/local/lib/ruby/3.1.0/bundler.rb:176:in `require'
#16 1.416 /app/config/application.rb:19:in `<main>'
#16 1.416 /app/Rakefile:4:in `require_relative'
#16 1.416 /app/Rakefile:4:in `<main>'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:29:in `load'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:29:in `load_rakefile'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:710:in `raw_load_rakefile'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:104:in `block in load_rakefile'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:103:in `load_rakefile'
#16 1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
#16 1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
#16 1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:18:in `perform'
#16 1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/command.rb:51:in `invoke'
#16 1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'
#16 1.416 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.416 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
#16 1.416 bin/rails:4:in `<main>'
#16 ERROR: process "/bin/sh -c bundle exec rails assets:precompile --trace --verbose" did not complete successfully: exit code: 1
------
 > [11/12] RUN bundle exec rails assets:precompile --trace --verbose:
1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:103:in `load_rakefile'
1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
1.416 /app/vendor/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:18:in `perform'
1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/command.rb:51:in `invoke'
1.416 /app/vendor/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'
1.416 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
1.416 /app/vendor/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
1.416 bin/rails:4:in `<main>'
------
Dockerfile:39
--------------------
  37 |       chmod 777 /app/bin/post-start
  38 |
  39 | >>> RUN bundle exec rails assets:precompile --trace --verbose
  40 |     RUN rm -rf /app/node_modules storage /usr/local/share/.cache/yarn log/* *.md test kube frontend spec tmp/cache lib/assets spec && \
  41 |       rm -rf /var/cache/apk/* && \
--------------------
ERROR: failed to solve: process "/bin/sh -c bundle exec rails assets:precompile --trace --verbose" did not complete successfully: exit code: 1

If you are able to achieve a local deployment of the service please let me know

astappiev commented 5 months ago

Hi @MDeLuise,

Yes, I have spent a couple of hours deploying the Trefle server, however, I gave up because of some other error. Unfortunately, I have no experience with Ruby at all.

Also, the database they provided as a dump is in plain CSV format. It will not be possible to import it into PostgreSQL, without pain (of course it's not the only one table).

So the easier way would be to crawl their API (it's still working, only search doesn't work, perhaps Redis or some microservice required for it is dead). I'm halfway though it, and will reach you back when it's done (I already fetched all species via /api/v1/species and now doing the second round via /api/v1/species/{id} to add more details).