search

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.97k stars 224 forks source link

feat: add dictionary encoding #3134

Open broccoliSpicy opened 3 days ago

broccoliSpicy commented 3 days ago

This PR tries to support dictionary encoding by integrating it with MiniBlock PageLayout.

The general approach here is: In a MiniBlock PageLayout, there is a optional dictionary field that stores a dictionary encoding if this miniblock has a dictionary.

/// A layout used for pages where the data is small
///
/// In this case we can fit many values into a single disk sector and transposing buffers is
/// expensive.  As a result, we do not transpose the buffers but compress the data into small
/// chunks (called mini blocks) which are roughly the size of a disk sector.
message MiniBlockLayout {
  // Description of the compression of repetition levels (e.g. how many bits per rep)
  ArrayEncoding rep_compression = 1;
  // Description of the compression of definition levels (e.g. how many bits per def)
  ArrayEncoding def_compression = 2;
  // Description of the compression of values
  ArrayEncoding value_compression = 3;
  ArrayEncoding dictionary = 4;
}

The rational for this is that if we dictionary encoding something, it's indices will definitely fall into a MiniBlockLayout. By doing this, we don't need to have a specific DictionaryEncoding, it can be any ArrayEncoding. The Dictionary and the indices are cascaded into another encoding automatically.

Currently, the dictionary is stored inside the page along with chunk meta data and chunk data, this is not ideal and is a TODO task.

This is a draft for discussion with the above idea so I only supported FixedWidthDataBlock with this encoding, the effort to add support for VariableWidthData is trivial.

3123

some performance comparison with parquet(no snappy), tpch dataset with scale factor 10. when set cache_bytes_per_column = 8 * 1024 * 1024: table name Parquet Read Time Lance Read Time Parquet File Size Lance File Size
customer 0.45s 0.42s 120 MiB 152 MiB
lineitem 11.55s 11.41s 2,076 MiB 3,374 MiB
orders 2.93s 3.60s 559 MiB 894 MiB
part 0.34s 0.41s 61 MiB 127 MiB
partsupp 3.29s 1.56s 401 MiB 470 MiB
Column Name DataType Parquet Read Time Lance Read Time Parquet File Size Lance File Size Cardinality
l_orderkey int32 1.57s 0.45s 181 MiB 178 MiB 15000000
l_partkey int32 1.65s 0.43s 261 MiB 151 MiB 2000000
l_suppkey int32 1.52s 0.36s 143 MiB 122 MiB
l_linenumber int32 0.27s 0.20s 13 MiB 22 MiB
l_quantity decimal128(15, 2) 2.10s 0.28s 43 MiB 46 MiB 50
l_extendedprice decimal128(15, 2) 5.10s 2.43s 318 MiB 917 MiB 1351462
l_discount decimal128(15, 2) 2.68s 0.15s 28 MiB 29 MiB 11
l_tax decimal128(15, 2) 2.63s 0.18s 28 MiB 29 MiB 9
l_returnflag string 1.32s 1.73s 13 MiB 15 MiB 3
l_linestatus string 1.98s 1.67s 8 MiB 8 MiB 2
l_shipdate date32[day] 1.18s 0.35s 86 MiB 101 MiB 2526
l_commitdate date32[day] 1.25s 0.33s 86 MiB 101 MiB 2466
l_receiptdate date32[day] 1.21s 0.35s 86 MiB 101 MiB 2555
l_shipinstruct string 2.03s 1.44s 14 MiB 15 MiB 4
l_shipmode string 1.69s 1.63s 21 MiB 22 MiB 7
l_comment string 6.03s 4.19s 738 MiB 1,505 MiB 7920544
o_orderkey int32 0.67s 0.30s 65MiB 44 MiB 15000000
o_custkey int32 0.35s 0.13s 65 MiB 37 MiB 999982
o_orderstatus string 0.46s 0.44s 3 MiB 3 MiB 3
o_totalprice decimal128(15, 2) 1.50s 0.71s 87 MiB 229 MiB 11944103
o_orderdate date32[day] 0.31s 0.18s 21 MiB 25 MiB 2406
o_orderpriority string 0.71s 0.37s 5 MiB 5 MiB 5
o_clerk string 0.76s 0.87s 25 MiB 212 MiB 10000
o_shippriority int32 0.04s 0.27s 0 MiB 57 MiB 1
o_comment string 2.88s 0.99s 284 MiB 277 MiB 7922330
when set cache_bytes_per_column = 32 * 1024 * 1024: table name Parquet Read Time Lance Read Time Parquet File Size Lance File Size
customer 0.45s 0.37s 120 MiB 134 MiB
lineitem 11.55s 8.71s 2,076 MiB 2,633 MiB
orders 2.93s 3.13s 559 MiB 795 MiB
part 0.34s 0.33s 61 MiB 107 MiB
partsupp 3.29s 1.71s 401 MiB 469 MiB
Column Name Parquet Read Time Lance Read Time Parquet File Size Lance File Size Cardinality
l_orderkey 1.57s 0.45s 181 MiB 178 MiB 15000000
l_partkey 1.65s 0.43s 261 MiB 151 MiB 2000000
l_suppkey 1.52s 0.36s 143 MiB 122 MiB 100000
l_linenumber 0.27s 0.19s 13 MiB 22 MiB 7
l_quantity 2.10s 0.36s 43 MiB 46 MiB 50
l_extendedprice 5.10s 2.16s 318 MiB 917 MiB 1351462
l_discount 2.68s 0.23s 28 MiB 29 MiB 11
l_tax 2.63s 0.14s 28 MiB 29 MiB 9
l_returnflag 1.32s 2.05s 13 MiB 15 MiB 3
l_linestatus 1.98s 2.11s 8 MiB 8 MiB 2
l_shipdate 1.18s 0.43s 86 MiB 101 MiB 2526
l_commitdate 1.25s 0.43s 86 MiB 101 MiB 2466
l_receiptdate 1.21s 0.41s 86 MiB 101 MiB 2555
l_shipinstruct 2.03s 1.86s 14 MiB 15 MiB 4
l_shipmode 1.69s 1.96s 21 MiB 22 MiB 7
l_comment 6.03s 2.12s 738 MiB 774 MiB 7920544
o_orderkey 0.67s 0.29s 65MiB 44 MiB 15000000
o_custkey 0.35s 0.27s 65 MiB 37 MiB 999982
o_orderstatus 0.46s 0.52s 3 MiB 3 MiB 3
o_totalprice 1.50s 0.55s 87 MiB 229 MiB 11944103
o_orderdate 0.31s 0.18s 21 MiB 25 MiB 2406
o_orderpriority 0.71s 0.52s 5 MiB 5 MiB 5
o_clerk 0.76s 0.68s 25 MiB 115 MiB 10000
o_shippriority 0.04s 0.16s 0 MiB 57 MiB 1
o_comment 2.88s 0.75s 284 MiB 275 MiB 7922330

to reproduce, here are the test scripts:

import pyarrow as pa
import pyarrow.parquet as pq
import datetime
import os
from lance.file import LanceFileReader, LanceFileWriter
import pandas as pd

# Directory paths for input and output files
parquet_folder = "/home/x/tpch/tpch_sf10/"
lance_folder = "/home/x/tpch/tpch_sf10/"

# Ensure output folders exist
os.makedirs(parquet_folder, exist_ok=True)
os.makedirs(lance_folder, exist_ok=True)

# Specific table and column to process
# table_name = "lineitem"
table_name = "orders"
column_to_process = "o_comment"
# Function to format throughput
def format_throughput(value):
    return f"{value:.2f} GiB/s"

# Function to flush file cache
def flush_file_cache():
    os.system('sync; echo 3 | sudo tee /proc/sys/vm/drop_caches')

# Batch size for reading
batch_size = 32 * 1024

# Processing only the l_comment column in the lineitem table
print(f"Processing table: {table_name} (column: {column_to_process})")

# File paths
parquet_file_path = os.path.join(parquet_folder, f"{table_name}.parquet")
lance_file_path = os.path.join(lance_folder, f"{table_name}_{column_to_process}.lance")
# lance_file_path = os.path.join(lance_folder, f"{table_name}_ps_comment.lance")

# Read only the l_comment column from Parquet
table_parquet = pq.read_table(parquet_file_path, columns=[column_to_process])
print(f"{table_name} rows: {table_parquet.num_rows}")

# Write to Parquet and benchmark write time
parquet_file_path = os.path.join(parquet_folder, f"{table_name}_{column_to_process}.parquet")
start = datetime.datetime.now()
pq.write_table(table_parquet, parquet_file_path)
end = datetime.datetime.now()
elapsed_parquet_write = (end - start).total_seconds()
print(f"Parquet write time for {table_name}, {column_to_process}: {elapsed_parquet_write:.2f}s")

# Write to Lance and benchmark write time
start = datetime.datetime.now()
with LanceFileWriter(lance_file_path, version="2.1") as writer:
    writer.write_batch(table_parquet)
end = datetime.datetime.now()
elapsed_lance_write = (end - start).total_seconds()
print(f"Lance write time for {table_name}, {column_to_process}: {elapsed_lance_write:.2f}s")

# Flush file cache
flush_file_cache()

# Benchmark read time for Parquet file
start = datetime.datetime.now()
parquet_file = pq.ParquetFile(parquet_file_path)
batches = parquet_file.iter_batches(batch_size=batch_size)
tab_parquet = pa.Table.from_batches(batches)
end = datetime.datetime.now()
elapsed_parquet_read = (end - start).total_seconds()
print(f"Parquet read time for {table_name}, {column_to_process}: {elapsed_parquet_read:.2f}s")

# Flush file cache again before reading Lance file
flush_file_cache()

# Benchmark read time for Lance file
start = datetime.datetime.now()
tab_lance = LanceFileReader(lance_file_path).read_all(batch_size=batch_size).to_table()
end = datetime.datetime.now()
elapsed_lance_read = (end - start).total_seconds()
print(f"Lance read time for {table_name}, {column_to_process}: {elapsed_lance_read:.2f}s")

# Compute total memory size
parquet_memory_size = tab_parquet.get_total_buffer_size()
lance_memory_size = tab_lance.get_total_buffer_size()

# Convert memory size to GiB
parquet_memory_size_gib = parquet_memory_size / (1024 * 1024 * 1024)
lance_memory_size_gib = lance_memory_size / (1024 * 1024 * 1024)

# Compute read throughput in GiB/sec
throughput_parquet_gib = parquet_memory_size_gib / elapsed_parquet_read
throughput_lance_gib = lance_memory_size_gib / elapsed_lance_read

# Format throughput values
formatted_throughput_parquet = format_throughput(throughput_parquet_gib)
formatted_throughput_lance = format_throughput(throughput_lance_gib)

# Output results
print(f"Parquet read throughput for {table_name}: {formatted_throughput_parquet}")
print(f"Lance read throughput for {table_name}: {formatted_throughput_lance}")

# Check file sizes
lance_file_size = os.path.getsize(lance_file_path)
lance_file_size_mib = lance_file_size // 1048576
parquet_file_size = os.path.getsize(parquet_file_path)
parquet_file_size_mib = parquet_file_size // 1048576

print(f"Parquet file size for {table_name}: {parquet_file_size} bytes ({parquet_file_size_mib:,} MiB)")
print(f"Lance file size for {table_name}: {lance_file_size} bytes ({lance_file_size_mib:,} MiB)")

# Verify data consistency between Parquet and Lance
assert tab_parquet == tab_lance, f"Data mismatch in table {table_name}"

print(f"Completed processing for table: {table_name}\n")
import pyarrow as pa
import pyarrow.parquet as pq
import datetime
import os
from lance.file import LanceFileReader, LanceFileWriter
import pandas as pd

# Directory paths for input and output files
parquet_folder = "/home/x/tpch/tpch_sf10/"
lance_folder = "/home/x/tpch/tpch_sf10/"

# Ensure output folders exist
os.makedirs(parquet_folder, exist_ok=True)
os.makedirs(lance_folder, exist_ok=True)

# List of TPC-H table names
# tables = ["partsupp"]
tables = ["customer", "lineitem", "nation", "orders", "part", "partsupp", "region", "supplier"]

# Function to format throughput
def format_throughput(value):
    return f"{value:.2f} GiB/s"

# Function to flush file cache
def flush_file_cache():
    os.system('sync; echo 3 | sudo tee /proc/sys/vm/drop_caches')

# Batch size for reading
batch_size = 32 * 1024

# Benchmark each TPC-H table
for table in tables:
    print(f"Processing table: {table}")

    # File paths
    parquet_file_path = os.path.join(parquet_folder, f"{table}.parquet")
    lance_file_path = os.path.join(lance_folder, f"{table}.lance")

    # Read the table from Parquet
    table_parquet = pq.read_table(parquet_file_path)
    print(f"{table} rows: {table_parquet.num_rows}")

    # Write to Parquet and benchmark write time
    start = datetime.datetime.now()
    pq.write_table(table_parquet, parquet_file_path)
    end = datetime.datetime.now()
    elapsed_parquet_write = (end - start).total_seconds()
    print(f"Parquet write time for {table}: {elapsed_parquet_write:.2f}s")

    # Write to Lance and benchmark write time
    start = datetime.datetime.now()
    with LanceFileWriter(lance_file_path, version = "2.1") as writer:
        writer.write_batch(table_parquet)
    end = datetime.datetime.now()
    elapsed_lance_write = (end - start).total_seconds()
    print(f"Lance write time for {table}: {elapsed_lance_write:.2f}s")

    # Flush file cache
    flush_file_cache()

    # Benchmark read time for Parquet file
    start = datetime.datetime.now()
    parquet_file = pq.ParquetFile(parquet_file_path)
    batches = parquet_file.iter_batches(batch_size=batch_size)
    tab_parquet = pa.Table.from_batches(batches)
    end = datetime.datetime.now()
    elapsed_parquet_read = (end - start).total_seconds()
    print(f"Parquet read time for {table}: {elapsed_parquet_read:.2f}s")

    # Flush file cache again before reading Lance file
    flush_file_cache()

    # Benchmark read time for Lance file
    start = datetime.datetime.now()
    tab_lance = LanceFileReader(lance_file_path).read_all(batch_size=batch_size).to_table()
    end = datetime.datetime.now()
    elapsed_lance_read = (end - start).total_seconds()
    print(f"Lance read time for {table}: {elapsed_lance_read:.2f}s")

    # Compute total memory size
    parquet_memory_size = tab_parquet.get_total_buffer_size()
    lance_memory_size = tab_lance.get_total_buffer_size()

    # Convert memory size to GiB
    parquet_memory_size_gib = parquet_memory_size / (1024 * 1024 * 1024)
    lance_memory_size_gib = lance_memory_size / (1024 * 1024 * 1024)

    # Compute read throughput in GiB/sec
    throughput_parquet_gib = parquet_memory_size_gib / elapsed_parquet_read
    throughput_lance_gib = lance_memory_size_gib / elapsed_lance_read

    # Format throughput values
    formatted_throughput_parquet = format_throughput(throughput_parquet_gib)
    formatted_throughput_lance = format_throughput(throughput_lance_gib)

    # Output results
    print(f"Parquet read throughput for {table}: {formatted_throughput_parquet}")
    print(f"Lance read throughput for {table}: {formatted_throughput_lance}")

    # Check file sizes
    lance_file_size = os.path.getsize(lance_file_path)
    lance_file_size_mib = lance_file_size // 1048576
    parquet_file_size = os.path.getsize(parquet_file_path)
    parquet_file_size_mib = parquet_file_size // 1048576

    print(f"Parquet file size for {table}: {parquet_file_size} bytes ({parquet_file_size_mib:,} MiB)")
    print(f"Lance file size for {table}: {lance_file_size} bytes ({lance_file_size_mib:,} MiB)")

    # Verify data consistency between Parquet and Lance
    assert tab_parquet == tab_lance, f"Data mismatch in table {table}"

    print(f"Completed processing for table: {table}\n")

print("All tables processed successfully.")
codecov-commenter commented 1 day ago

Codecov Report

Attention: Patch coverage is 19.14894% with 266 lines in your changes missing coverage. Please review.

Project coverage is 77.69%. Comparing base (1d3b204) to head (0a6f6c9).

Files with missing lines Patch % Lines
.../lance-encoding/src/encodings/logical/primitive.rs 13.10% 172 Missing and 7 partials :warning:
...st/lance-encoding/src/encodings/physical/binary.rs 0.00% 59 Missing :warning:
rust/lance-encoding/src/encoder.rs 0.00% 14 Missing :warning:
rust/lance-core/src/utils/hash.rs 0.00% 6 Missing :warning:
rust/lance-encoding/src/format.rs 28.57% 5 Missing :warning:
rust/lance-encoding/src/decoder.rs 0.00% 1 Missing :warning:
rust/lance-encoding/src/encodings/physical.rs 0.00% 1 Missing :warning:
rust/lance-encoding/src/statistics.rs 97.14% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3134 +/- ## ========================================== - Coverage 77.95% 77.69% -0.26% ========================================== Files 242 243 +1 Lines 81904 82206 +302 Branches 81904 82206 +302 ========================================== + Hits 63848 63874 +26 - Misses 14890 15152 +262 - Partials 3166 3180 +14 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3134/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `77.69% <19.14%> (-0.26%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

broccoliSpicy commented 12 hours ago
after make append method in DataBlockBuilderImpl use immutable borrow: table name Parquet Read Time Lance Read Time Parquet File Size Lance File Size
customer 0.45s 0.42s 120 MiB 152 MiB
lineitem 11.55s 10.46s 2,076 MiB 3,374 MiB
orders 2.93s 2.78s 559 MiB 894 MiB
part 0.34s 0.40s 61 MiB 127 MiB
partsupp 3.29s 1.89s 401 MiB 470 MiB
Column Name DataType Parquet Read Time Lance Read Time Parquet File Size Lance File Size Cardinality
p_partkey int32 0.05s 0.02s 8 MiB 4 MiB 2000000
p_name string 0.45s 0.09s 25 MiB 30 MiB 1999828
p_mfgr string 0.07s 0.03s 0.7 MiB 0.7 MiB 5
p_brand string 0.06s 0.03s 1 MiB 1 MiB 25
p_type string 0.10s 0.23s 1 MiB 38 MiB 150
p_size int32 0.02s 0.02s 1 MiB 1 MiB 50
p_container string 0.08s 0.03s 1 MiB 1 MiB 40
p_retailprice decimal128(15, 2) 0.11s 0.20 s 3 MiB 30 MiB 31681
p_comment string 0.32s 0.19s 16 MiB 27 MiB 754704
broccoliSpicy commented 9 hours ago

First 100 rows: l_extendedprice [[Decimal('33078.94')] [Decimal('38306.16')] [Decimal('15479.68')] [Decimal('34616.68')] [Decimal('28974.00')] [Decimal('44842.88')] [Decimal('63066.32')] [Decimal('86083.65')] [Decimal('70822.15')] [Decimal('39620.34')] [Decimal('3581.56')] [Decimal('52411.80')] [Decimal('35032.14')] [Decimal('39819.00')] [Decimal('25179.60')] [Decimal('31387.20')] [Decimal('68864.50')] [Decimal('53697.73')] [Decimal('17273.04')] [Decimal('12423.15')] [Decimal('84904.50')] [Decimal('46245.92')] [Decimal('74398.68')] [Decimal('55806.45')] [Decimal('7216.50')] [Decimal('26963.72')] [Decimal('40995.52')] [Decimal('3091.16')] [Decimal('5393.68')] [Decimal('46642.64')] [Decimal('6978.84')] [Decimal('39224.92')] [Decimal('34948.80')] [Decimal('8803.10')] [Decimal('49780.56')] [Decimal('20768.41')] [Decimal('24817.98')] [Decimal('8558.10')] [Decimal('33708.00')] [Decimal('44788.54')] [Decimal('13026.23')] [Decimal('42317.50')] [Decimal('42877.74')] [Decimal('45516.80')] [Decimal('74029.62')] [Decimal('48691.20')] [Decimal('69449.25')] [Decimal('45538.29')] [Decimal('63681.20')] [Decimal('49288.36')] [Decimal('46194.72')] [Decimal('58892.42')] [Decimal('57788.48')] [Decimal('52982.88')] [Decimal('68665.20')] [Decimal('30837.66')] [Decimal('52933.66')] [Decimal('26050.42')] [Decimal('37545.27')] [Decimal('37916.72')] [Decimal('78670.80')] [Decimal('5069.36')] [Decimal('21910.92')] [Decimal('10159.55')] [Decimal('48887.96')] [Decimal('23784.30')] [Decimal('33001.13')] [Decimal('4925.01')] [Decimal('84764.66')] [Decimal('84721.88')] [Decimal('26424.60')] [Decimal('40541.31')] [Decimal('46006.50')] [Decimal('63853.40')] [Decimal('54433.44')] [Decimal('55447.68')] [Decimal('29539.20')] [Decimal('3279.00')] [Decimal('72225.30')] [Decimal('25852.69')] [Decimal('9761.92')] [Decimal('20974.98')] [Decimal('1186.00')] [Decimal('14182.41')] [Decimal('50996.73')] [Decimal('30371.88')] [Decimal('30631.75')] [Decimal('3330.36')] [Decimal('61348.50')] [Decimal('49876.20')] [Decimal('57583.11')] [Decimal('47574.50')] [Decimal('38862.87')] [Decimal('58554.90')] [Decimal('24241.36')] [Decimal('61777.05')] [Decimal('39272.24')] [Decimal('29739.92')] [Decimal('1424.37')] [Decimal('14056.42')]]

first 100 rows p_retailprice [[Decimal('901.00')] [Decimal('902.00')] [Decimal('903.00')] [Decimal('904.00')] [Decimal('905.00')] [Decimal('906.00')] [Decimal('907.00')] [Decimal('908.00')] [Decimal('909.00')] [Decimal('910.01')] [Decimal('911.01')] [Decimal('912.01')] [Decimal('913.01')] [Decimal('914.01')] [Decimal('915.01')] [Decimal('916.01')] [Decimal('917.01')] [Decimal('918.01')] [Decimal('919.01')] [Decimal('920.02')] [Decimal('921.02')] [Decimal('922.02')] [Decimal('923.02')] [Decimal('924.02')] [Decimal('925.02')] [Decimal('926.02')] [Decimal('927.02')] [Decimal('928.02')] [Decimal('929.02')] [Decimal('930.03')] [Decimal('931.03')] [Decimal('932.03')] [Decimal('933.03')] [Decimal('934.03')] [Decimal('935.03')] [Decimal('936.03')] [Decimal('937.03')] [Decimal('938.03')] [Decimal('939.03')] [Decimal('940.04')] [Decimal('941.04')] [Decimal('942.04')] [Decimal('943.04')] [Decimal('944.04')] [Decimal('945.04')] [Decimal('946.04')] [Decimal('947.04')] [Decimal('948.04')] [Decimal('949.04')] [Decimal('950.05')] [Decimal('951.05')] [Decimal('952.05')] [Decimal('953.05')] [Decimal('954.05')] [Decimal('955.05')] [Decimal('956.05')] [Decimal('957.05')] [Decimal('958.05')] [Decimal('959.05')] [Decimal('960.06')] [Decimal('961.06')] [Decimal('962.06')] [Decimal('963.06')] [Decimal('964.06')] [Decimal('965.06')] [Decimal('966.06')] [Decimal('967.06')] [Decimal('968.06')] [Decimal('969.06')] [Decimal('970.07')] [Decimal('971.07')] [Decimal('972.07')] [Decimal('973.07')] [Decimal('974.07')] [Decimal('975.07')] [Decimal('976.07')] [Decimal('977.07')] [Decimal('978.07')] [Decimal('979.07')] [Decimal('980.08')] [Decimal('981.08')] [Decimal('982.08')] [Decimal('983.08')] [Decimal('984.08')] [Decimal('985.08')] [Decimal('986.08')] [Decimal('987.08')] [Decimal('988.08')] [Decimal('989.08')] [Decimal('990.09')] [Decimal('991.09')] [Decimal('992.09')] [Decimal('993.09')] [Decimal('994.09')] [Decimal('995.09')] [Decimal('996.09')] [Decimal('997.09')] [Decimal('998.09')] [Decimal('999.09')] [Decimal('1000.10')]]