apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
5.86k stars 2.06k forks source link

catalog issue #10324

Open Shantanu0701 opened 1 month ago

Shantanu0701 commented 1 month ago

Below code:

import pyspark
from pyspark.sql import SparkSession
conf = (
    pyspark.SparkConf()
       .setAppName('app_name')  # Set the application name for Spark
        # Specify the Iceberg package to be used
       .set('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.1')
        # Add Iceberg SQL Extensions to enable Iceberg-specific SQL operations
       .set('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
        # Configure the Iceberg catalog
       .set('spark.sql.catalog.localcat', 'org.apache.iceberg.spark.SparkCatalog')  # Define the catalog implementation
       .set('spark.sql.catalog.localcat.type', 'hadoop')  # Set the catalog type to Hadoop
       .set('spark.sql.catalog.localcat.warehouse', '/home/user/Downloads/warehouse')  # Specify the warehouse directory
)

Create a SparkSession with the previously defined configurations

spark = SparkSession.builder.config(conf=conf).getOrCreate()
print("Spark Running")

Create a table named 'first_table' in the 'localcat.localdb' catalog using Iceberg format

spark.sql("CREATE TABLE localcat.localdb.first_table (id bigint, first_name string, last_name string) USING iceberg")

ISSUE: So basically my table is under localcat catalog and localdb database. But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.

nastra commented 1 month ago

But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.

Do you have example output that describe what you're seeing vs what you're expecting here?