search

XRPLF / clio

An XRP Ledger API Server
https://xrpl.org
ISC License
61 stars 52 forks source link

Can't process new ledgers: The current ETL source is not compatible with the version of the libxrpl Clio is currently using. Please upgrade Clio to a newer version (Version: 2.1.1~26ed78f-1) #1410

Closed PaulMoney closed 1 month ago

PaulMoney commented 4 months ago

Issue Description

Hello, currently, I'm running my own small private rippled chain where I'm testing a new rippled plugin feature with a custom transaction type (e.g., TokenSwap). However, when I want to connect Clio to one of my custom rippled nodes, I receive the following error:

2024-05-15 10:44:00.555862 (etl/impl/AmendmentBlock.hpp:39) [0x00007f8c97fff700] ETL:FTL Can't process new ledgers: The current ETL source is not compatible with the version of the libxrpl Clio is currently using. Please upgrade Clio to a newer version.

I have upgraded Clio to the newest version available as Debian package (2.1.1~26ed78f-1), but the error persists. Also, what I have noticed, is that, if I manually specify start_sequence field in Clio configuration file, it starts extracting ledgers, however on some ledger were was TokenSwap transaction it stops and Clio reports that ledger with that index could not be found.

Is it possible to rebuild or include support for custom transaction types in Clio?

Steps to Reproduce

  1. Build rippled node/nodes from this commit - https://github.com/mvadari/rippled/tree/d8e0cd1e6f51b6cdb91a99c4213afaaa35e968b8
  2. Build xrpl-plugin, as noted here - https://github.com/mvadari/xrpl-plugin/tree/main/python
  3. Send custom transaction (e.g., TokenSwap) to the rippled network.

Expected Result

Clio connects to custom rippled node and saves custom rippled transaction (e.g., TokenSwap).

Actual Result

Clio is unable to process new ledgers because current ETL source is not compatible with the version of the libxrpl Clio is currently using.

Environment

Ubuntu 22.04

godexsoft commented 4 months ago

Hi Paul,

Clio can't proceed with its ETL side of things if some unknown type arrives from rippled. Is the custom type added in the plugin (in python) or in C++ code? I don't know enough about the plugin support so thought i'd ask.

In order to use a custom rippled (or really, libxrpl):

The workflow above is basically what is usually required when working on something that is not in libxrpl already. There is an issue to improve the docs with the info above.

Please keep in mind that if you change the rippled code you need to conan export . again each time to avoid using the previous version and running into issues. This is the current downside of the workflow i described.

If your custom type is in python only, if that's even a possibility at all, then the above will not help. And frankly i don't see a way to even make it work. But hopefully new types can't be added outside of C++.

Hope this helps. Let us know if you need more help to get it to work 👍

PaulMoney commented 4 months ago

Hey @godexsoft Followed your steps in building custom Clio server, however, when I ran this command:

cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release ..

I received the following error - 

CMake Error at src/util/CMakeLists.txt:27 (target_link_libraries):
  Target "clio_util" links to:

    xrpl::libxrpl

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

P.s. The first command - 

conan install .. --output-folder . --build missing --settings build_type=Release -o tests=True -o lint=False

Works correctly without any issues.

Maybe I need to edit something else ?

godexsoft commented 4 months ago

Hi @PaulMoney, I think you also need to change xrpl:: to xrpl-custom:: (or what you used for the name in the previous steps) in util/CMakeLists.txt. As you updated the name of the package, now it should also be reflected where we try and link to the library. I'll update the steps above for completeness too.

PaulMoney commented 4 months ago

@godexsoft Now, the problem is with this command - 

cmake --build . --parallel $(nproc)

When I run, I receive the following error - 

In file included from /home/hero/.conan/data/xrpl-custom/2.2.0-rc1/_/_/package/c22f53cf03124b247fb1cf8a08a7c409b4b2716e/include/ripple/protocol/TxFormats.h:23,
                 from /home/hero/clio/src/util/TxUtils.cpp:22:
/home/hero/.conan/data/xrpl-custom/2.2.0-rc1/_/_/package/c22f53cf03124b247fb1cf8a08a7c409b4b2716e/include/ripple/protocol/KnownFormats.h:25:10: fatal error: ripple/plugin/plugin.h: No such file or directory
   25 | #include <ripple/plugin/plugin.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

This error seems to be logical, as if I build rippled core from this branch - https://github.com/mvadari/rippled/tree/d8e0cd1e6f51b6cdb91a99c4213afaaa35e968b8

It has a new plugin folder, where plugin functionality is being placed.

Maybe you have any ideas how to workaround this ?

godexsoft commented 4 months ago

@PaulMoney I think this means that the plugin.h (and probably some others) are not actually in libxrpl but only in rippled itself. This would require to add them in the correct places in rippled's cmake files so these files are exported with the library.

Maybe @thejohnfreeman can give you some more concrete tips on this. I'm not too familiar with the build process of rippled.

thejohnfreeman commented 4 months ago

My suggestion: build both rippled and clio from source. The packages expect specific compatible versions as dependencies, but you want to use a custom version of rippled that is modified from an older version.

In your custom source tree for rippled:

In a source tree for a version of Clio compatible with the base version of rippled that you modified:

PaulMoney commented 4 months ago

@thejohnfreeman Thanks for your response.

Could you please explain the exact commands I should run in source tree for rippled?

For now, I ran only this command - conan export . xrpl/2.2.0-rc1@hero/custom, and when I build Clio I receive the same error - 

In file included from /home/hero/.conan/data/xrpl/2.2.0-rc1/hero/custom/package/5bb612f3ef45687cc5f5c326d3154007c5053ab8/include/ripple/protocol/TxFormats.h:23,
                 from /home/hero/clio/src/util/TxUtils.cpp:22:
/home/hero/.conan/data/xrpl/2.2.0-rc1/hero/custom/package/5bb612f3ef45687cc5f5c326d3154007c5053ab8/include/ripple/protocol/KnownFormats.h:25:10: fatal error: ripple/plugin/plugin.h: No such file or directory
   25 | #include <ripple/plugin/plugin.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
thejohnfreeman commented 3 months ago

Ok I walked through the process and found two problems in the rippled branch you are using:

  1. The Conan recipe is outdated.
  2. The plugin public headers are not installed in Builds/CMake/RippledCore.cmake. (This is the kind of oversight that will become impossible once https://github.com/XRPLF/rippled/pull/4997 is merged.)

I've fixed these problems in a branch in my fork. With that branch, the below script works:

# Need a build environment with Conan >= 1.60 and GCC >= 12. This is just one example.
sudo docker run --rm -it rippleci/clio_ci

# Check out the custom version of rippled you want to use.
git clone https://github.com/thejohnfreeman/rippled --branch plugins
pushd rippled

# These commands come from the build instructions for rippled.
# Conan is already configured in this container.
conan export external/snappy snappy/1.1.10@
conan export external/rocksdb rocksdb/6.29.5@
conan export external/soci soci/4.0.3@

# We don't have to build rippled yet. Just export it to local cache for Clio.
conan export . hero/custom
# Note the package reference you just exported.
reference="xrpl/2.2.0-b3@hero/custom"

popd
git clone https://github.com/XRPLF/clio.git
pushd clio

# Update the requirement to find the one we exported above.
sed -i "s|xrpl/2.2.0-rc3|${reference}|" conanfile.py

# These come from the build instructions for Clio.
mkdir .build
pushd .build
# There seems to be a dependency version conflict. Resolve it with `--require-override`.
conan install .. --output-folder . --build missing --settings build_type=Release -o tests=True -o lint=False --require-override zlib/1.2.13
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --parallel 8
mvadari commented 3 months ago

FYI I haven't done any work with Clio to make sure that it supports plugins; there's probably a fair amount of work on that front.

PaulMoney commented 3 months ago

Hey @thejohnfreeman Thanks for your tries to build Clio. I have followed your steps, however I received the following error when I ran the last build command:

[  9%] Building CXX object src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o
/home/hero/clio/src/util/TxUtils.cpp:22:10: fatal error: xrpl/protocol/TxFormats.h: No such file or directory
   22 | #include <xrpl/protocol/TxFormats.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
godexsoft commented 3 months ago

@PaulMoney i had the same issue yesterday when updating my local env to latest Clio develop - delete the build folder and start over from the conan command. Seems like your build is still setup to use the previous libxrpl.

Or if you are on a very custom libxrpl/rippled branch then you want to merge latest upstream libxrpl into your branch because they recently changed paths (ripple -> xrpl)