search

tableau / hyper-api-samples

Sample code to get started with the Hyper API.
https://help.tableau.com/current/api/hyper_api/en-us/index.html
MIT License
132 stars 71 forks source link

tableauhyperapi (v0.0.16123) HyperException: This database does not support 128-bit numerics. #81

Closed Capital-de-dev closed 5 months ago

Capital-de-dev commented 1 year ago 
test@local: pip3 list | grep hyper   
tableauhyperapi        0.0.16377 
What’s New in the Hyper API (v0.0.16123)
December 7, 2022

Added support for 128-bit numerics. This allows a precision of up to 38 for the NUMERIC SQL type.

Run:

from pathlib import Path

from tableauhyperapi import HyperProcess, Telemetry, \
    Connection, CreateMode, \
    NOT_NULLABLE, NULLABLE, SqlType, TableDefinition, \
    Inserter, \
    escape_name, escape_string_literal, \
    HyperException

customer_table = TableDefinition(
    table_name="Customer",
    columns=[
        TableDefinition.Column("Customer ID", SqlType.text(), NOT_NULLABLE),
        TableDefinition.Column("Customer Amount", SqlType.numeric(38,3), NOT_NULLABLE),
        TableDefinition.Column("Customer Score", SqlType.big_int(), NOT_NULLABLE)
    ]
)

path_to_database = Path("customer.hyper")

with HyperProcess(telemetry=Telemetry.SEND_USAGE_DATA_TO_TABLEAU) as hyper:

    connection_parameters = {"lc_time": "en_US"}

    with Connection(endpoint=hyper.endpoint,
                        database=path_to_database,
                        create_mode=CreateMode.CREATE_AND_REPLACE,
                        parameters=connection_parameters) as connection:

        connection.catalog.create_table(table_definition=customer_table)

Output:

---------------------------------------------------------------------------
HyperException                            Traceback (most recent call last)
Input In [32], in <module>
     23 connection_parameters = {"lc_time": "en_US"}
     25 with Connection(endpoint=hyper.endpoint,
     26                     database=path_to_database,
     27                     create_mode=CreateMode.CREATE_AND_REPLACE,
     28                     parameters=connection_parameters) as connection:
---> 30     connection.catalog.create_table(table_definition=customer_table)

File ~/.local/lib/python3.8/site-packages/tableauhyperapi/catalog.py:65, in Catalog.create_table(self, table_definition)
     59 def create_table(self, table_definition: TableDefinition):
     60     """
     61     Creates a table. Raise an exception if the table already exists.
     62 
     63     :param table_definition: the table definition.
     64     """
---> 65     self.__create_table(table_definition, True)

File ~/.local/lib/python3.8/site-packages/tableauhyperapi/catalog.py:57, in Catalog.__create_table(self, table_definition, fail_if_exists)
     55 def __create_table(self, table_definition: TableDefinition, fail_if_exists: bool):
     56     native_table_def = SchemaConverter.table_definition_to_native(table_definition)
---> 57     Error.check(hapi.hyper_create_table(self.__cdata, native_table_def.cdata, fail_if_exists))

File ~/.local/lib/python3.8/site-packages/tableauhyperapi/impl/dllutil.py:100, in Error.check(p)
     97 if p != ffi.NULL:
     98     # this will free the error when it goes out of scope
     99     errp = Error(p)
--> 100     raise errp.to_exception()

HyperException: This database does not support 128-bit numerics.
Context: 0xfa6b0e2f
vogelsgesang commented 1 year ago

Our release notes are confusing... We should fix that.

Hyper API is now able to read and operate on 128 bit numerics. I.e., if you get a 128 bit numeric e.g. from a Parquet file, you can use normal arithmetic on it. But before you store it into a Hyper file

However, there is not yet any Hyper file format version which allows you to write 128 bit numerics. Such a new file format version is already lined up to ship very soon :)

However, you also have another problem: There is not yet any Tableau version which is able to read 128 bit numerics from Hyper files... Soon 🙂

vogelsgesang commented 1 year ago

(Leaving this issue open until we fix our release notes, such that it can be found by others running into this, too)

miky-schreiber commented 10 months ago

@vogelsgesang Thank you for this answer, helped us a lot! Do we have an ETA for a version which will support writing 128 bit numerics into a Hyper file?

rajuvg commented 5 months ago

@vogelsgesang any updates about the version which will support writing 128 bit numeric to Hyper file?

vogelsgesang commented 5 months ago

In fact, there already is a database version 3, similar to the other database versions documented in https://tableau.github.io/hyper-db/docs/hyper-api/hyper_process#databasesettings This database version has been there for a long time already.

Hyper itself would be ready, you can already write database files with 128-bit numerics. The problem is rather: Will Tableau be able to open them? Afaik, more up-to-date versions of Tableau should already be able to do so...

vogelsgesang commented 5 months ago

We added documentation for file format 3. The documentation will be published as part of the next Hyper API release. In the meantime, you can take a sneak peak at it in https://github.com/tableau/hyper-db/pull/93/files

rajuvg commented 5 months ago

Thanks for the details, Could you please confirm hyperd version whichever supports 128 bit decimal and we re running tableau servers version as 2023.1.1

vogelsgesang commented 5 months ago

Quoting from the PR linked above:

The database file format version 3 is supported by Tableau Desktop 2022.4.1 and Server 2023.1 and newer product versions. It is supported by Tableau Prep 2022.4.1 and newer versions

Does this answer your question?

rajuvg commented 5 months ago

Sorry for the confusion what is the Hyped file version which is used for creating hyperprocess instance. Thaks Raju

On Mon, Apr 22, 2024, 3:07 PM Adrian Vogelsgesang @.***> wrote:

Quoting from the PR linked above:

The database file format version 3 is supported by Tableau Desktop 2022.4.1 and Server 2023.1 and newer product versions. It is supported by Tableau Prep 2022.4.1 and newer versions

Does this answer your question?

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2070862621, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCVCURYVZCKPWODYYALY6VUX7AVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQHA3DENRSGE . You are receiving this because you commented.Message ID: @.***>

vogelsgesang commented 5 months ago

Sorry, I don't understand the question.

The hyper file version used to create the HyperProcess instance only influences the files written by that Hyper process.

When it comes to reading (and I assume you are asking about which Tableau versions will be able to read those Hyper files?), then the file format version set as the default_database_version has no influence on whether Hyper can read this file or not

rajuvg commented 5 months ago

Tableau provided hyperd.exe for widows and hyperd for Linux so that we can create hyper process in java code and create hyper file. My question version if hyperd instance will allow us to create hyperfile with 128but decimal and publish it to tableau server2023.1.1. Hope my question is clear. Thanks in advance. Regards, Raju

On Mon, Apr 22, 2024, 3:55 PM Adrian Vogelsgesang @.***> wrote:

Sorry, I don't understand the question.

The hyper file version used to create the HyperProcess instance only influences the files written by that Hyper process.

When it comes to reading (and I assume you are asking about which Tableau versions will be able to read those Hyper files?), then the file format version set as the default_database_version has no influence on whether Hyper can read this file or not

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2070936996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCQYMNPKBPAA25ZVF33Y6V2MTAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQHEZTMOJZGY . You are receiving this because you commented.Message ID: @.***>

rajuvg commented 4 months ago

Thanks by setting the default_database_version:3 helped in publishing 128 bit data where as when I am doing the append it is failing..is this option supported. Thanks & Regards Raju

On Mon, Apr 22, 2024 at 6:46 PM raju vadali @.***> wrote:

Tableau provided hyperd.exe for widows and hyperd for Linux so that we can create hyper process in java code and create hyper file. My question version if hyperd instance will allow us to create hyperfile with 128but decimal and publish it to tableau server2023.1.1. Hope my question is clear. Thanks in advance. Regards, Raju

On Mon, Apr 22, 2024, 3:55 PM Adrian Vogelsgesang < @.***> wrote:

Sorry, I don't understand the question.

The hyper file version used to create the HyperProcess instance only influences the files written by that Hyper process.

When it comes to reading (and I assume you are asking about which Tableau versions will be able to read those Hyper files?), then the file format version set as the default_database_version has no influence on whether Hyper can read this file or not

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2070936996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCQYMNPKBPAA25ZVF33Y6V2MTAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQHEZTMOJZGY . You are receiving this because you commented.Message ID: @.***>

rajuvg commented 4 months ago

Currently we are encountering an issue if we publish a 128 bit hyperfile to Tableauserver2023.1 with override flag as false, can you confirm starting from which version of tableau server this option is supported.Thanks in advance. Regards, Raju

On Mon, Apr 29, 2024 at 9:13 PM raju vadali @.***> wrote:

Thanks by setting the default_database_version:3 helped in publishing 128 bit data where as when I am doing the append it is failing..is this option supported. Thanks & Regards Raju

On Mon, Apr 22, 2024 at 6:46 PM raju vadali @.***> wrote:

Tableau provided hyperd.exe for widows and hyperd for Linux so that we can create hyper process in java code and create hyper file. My question version if hyperd instance will allow us to create hyperfile with 128but decimal and publish it to tableau server2023.1.1. Hope my question is clear. Thanks in advance. Regards, Raju

On Mon, Apr 22, 2024, 3:55 PM Adrian Vogelsgesang < @.***> wrote:

Sorry, I don't understand the question.

The hyper file version used to create the HyperProcess instance only influences the files written by that Hyper process.

When it comes to reading (and I assume you are asking about which Tableau versions will be able to read those Hyper files?), then the file format version set as the default_database_version has no influence on whether Hyper can read this file or not

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2070936996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCQYMNPKBPAA25ZVF33Y6V2MTAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQHEZTMOJZGY . You are receiving this because you commented.Message ID: @.***>

vogelsgesang commented 4 months ago

Thanks by setting the default_database_version:3 helped in publishing 128 bit data where as when I am doing the append it is failing..is this option supported.

A database file is always using the version which was set during the original creation time of that file. To upgrade to a new version, you need to create a new Hyper file and copy over all your tables from the old file. See https://github.com/tableau/hyper-api-samples/blob/main/Community-Supported/convert-hyper-file/convert_hyper_file.py for an example of how to do so from Python

vogelsgesang commented 4 months ago

[...] if we publish a 128 bit hyperfile to Tableauserver2023.1 with override flag as false,

which exact API call are you using?

rajuvg commented 4 months ago

we are using url=String.format(,overrise=false); restTemplate.exchange(url,HTTPMethod.post,ent,String.class)); Please let me know if you need more information.

On Tue, Apr 30, 2024 at 8:05 AM Adrian Vogelsgesang < @.***> wrote:

[...] if we publish a 128 bit hyperfile to Tableauserver2023.1 with override flag as false,

which exact API call are you using?

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2085285805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCUXDWFHCU6FJC7UI3DY76JKVAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGI4DKOBQGU . You are receiving this because you commented.Message ID: @.***>

rajuvg commented 4 months ago

we are using url=String.format(,append=true); restTemplate.exchange(url,HTTPMethod.post,ent,String.class)); Please let me know if you need more information.

On Tue, Apr 30, 2024 at 8:50 AM raju vadali @.***> wrote:

we are using url=String.format(,overrise=false); restTemplate.exchange(url,HTTPMethod.post,ent,String.class)); Please let me know if you need more information.

On Tue, Apr 30, 2024 at 8:05 AM Adrian Vogelsgesang < @.***> wrote:

[...] if we publish a 128 bit hyperfile to Tableauserver2023.1 with override flag as false,

which exact API call are you using?

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2085285805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCUXDWFHCU6FJC7UI3DY76JKVAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGI4DKOBQGU . You are receiving this because you commented.Message ID: @.***>

vogelsgesang commented 4 months ago

I have no idea which Tableau REST API your two source code lines are using. Please provide a self-contained repro

rajuvg commented 4 months ago

Hi, I think we now know the problem tableau (2023.1 .1) is failing to open the hyperfile uploaded with default_database_version=3 as it has been published with append=true for merging in another hyper file previously published as override=true and default database.version =3 meaning tableau server having issue in merging to hyper files having default_database_version=3.Appreciate any help in overcoming this issue .

Thanks in Advance. Raju

On Tue, Apr 30, 2024 at 10:18 AM Adrian Vogelsgesang < @.***> wrote:

I have no idea which Tableau REST API your two source code lines are using. Please provide a self-contained repro

— Reply to this email directly, view it on GitHub https://github.com/tableau/hyper-api-samples/issues/81#issuecomment-2085629170, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7CDCSM26EPADAU3Z6NIL3Y76Y3JAVCNFSM6AAAAAAUBWQ436VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGYZDSMJXGA . You are receiving this because you commented.Message ID: @.***>