Closed GoogleCodeExporter closed 8 years ago
Pouria,
It seems the issue is in one of the file names that you are using.
Specifically, you have a typo in the name of the file number 27: "fbu27..adm"
which exists in the raninbow cluster.
As you can see you have one additional dot. If you remove it, the load succeed.
Now, why the system was in a hung state and didn't report an error, this is a
different issue. It seems the exception is swallowed somewhere.
Re-assigning this to Raman since this is now related to the external data scan
operator.
Original comment by salsuba...@gmail.com
on 19 Jun 2013 at 6:34
Sattam,
That is just a typo, that happened when I copied the DDL to this issue. If you
fix that *typo*, again it is not working. For that typo system reports the
error properly as "file not found" and there is no hung.
Original comment by pouria.p...@gmail.com
on 19 Jun 2013 at 7:00
Copying this from the email thread for documentation purpose:
Then I'm not able to reproduce it.
When I fix the typo, the load succeed. I'm using the same settings that you
indicated in the issue description (7 NCs, 4 IO devices, and 28 file splits).
Actually, I'm using the same xml configuration file except changing the paths.
Original comment by salsuba...@gmail.com
on 19 Jun 2013 at 7:13
BTW, the typo is in the actual file name (the one that exists in the rainbow
cluster in this location: /data/pouria/load-issue-data/fbu/fbu27..adm) and not
the DDL statement. The DDL statement has no typos.
Are we on the same page?
Original comment by salsuba...@gmail.com
on 19 Jun 2013 at 7:16
Based on comments
#1, #2
and last observation from Jarod:-
> This time all machines have enough space for the data. Each machine have
> 2.7TB available space. The experiment starts 9:48am, and since now the
> loading is still on-going, but all node controllers have no CPU activity (so
> I assume that the load is already failed). I tried to get the lines of the
> dataset to be updated, and it returns 0.
it doesn't seem to be an issue with the path specified in the AQL. As reported,
(i) correcting the path does not get rid of the issue.
(ii) incorrect path leads to exception stating "invalid path"
(iii) having sufficient space on disk still causes the issue.
(iii) the issue is consistently reproducible on a single node and on asterix-cluster
@Sattam, I believe the assertion in #1 is invalid. I would recommend debugging
it on either the cluster or pouria's machine where the issue is consistently
reproducible.
I am sensing another issue here:- if the path is invalid, load statement does
not give an exception but runs into a hung state. I am investigating the load
statement with regards to incorrect paths.
Original comment by RamanGro...@gmail.com
on 19 Jun 2013 at 7:23
OK, here is what it is happening:
Pouria is using an accurate estimate of the dataset cardinailty which is 337M
records. This hint is used by the indexes to create the bloom filters. Each
record approximately requires 10 bits. Thus the total sizes of the bloom
filters will be around 400MB. Note that all the pages of the bloom filters will
be pinned in the buffer cache while the index is being loaded or open. The
current default size of the buffer cache in the XML configuration file is 32
MB :-)
Now the issue is that we over estimate the size of the dataset size in each
partition by using the total dataset cardinality which is an over kill. Thus if
we have 7 NCs where each NC has 4 IO devices, each bloom filter will be of size
400MB where it should be 14MB.
Obviously 400MB >> 32MB
Which will make the buffer cache spin looking for a victim page to flush to
disk, but it cannot find any; therefore, the bulkload cannot progress.
To summarize:
1) We should use a better estimate for the cardinality of the dataset per
partition based on number of NCs and number of IO devices.
2) We should change the default size of the buffer cache from 32MB to a higher
value.
Original comment by salsuba...@gmail.com
on 20 Jun 2013 at 11:19
Original comment by salsuba...@gmail.com
on 21 Jun 2013 at 4:58
I think we have a space management "design flaw" here. IMO we should not put
these bits in the buffer pool - the buffer pool is intended to serve as a cache
for pages from disk that can come in and go out more fluidly. These bits are
analogous (again IMO) to the in-memory components, and should come out of that
storage pool - part of the memory "price" of an LSM index is its in-memory
component, which is pinned for the duration while it's open, and this seems
like another memory price component.
Original comment by dtab...@gmail.com
on 22 Jun 2013 at 8:23
Well, it depends on how you look at it.
The purpose of the bloom filter is to avoid going to disk when answering a
query by doing caching. The bloom filter pages will be kicked out from the
buffer cache when the index is closed. As long as the index is open, then
the assumption is that the index is going to be searched, thus we keep the
filter's pages pinned in the cache.
So I don't really see how this break the purpose of the buffer pool.
Besides that, if you think about it, the in memory buffer cache is not
really a cache, in the sense that we maintain inserts in the in memory for
the purpose of having high write throughput and not for the purpose of
serving search queries faster. While the bloom filter purpose is totally
different, it is used to make search queries run faster.
That being said, as Vinayak pointed out yesterday, we should have the
ability to do something when we hit this scenario, by extending the size of
the buffer cache, or maybe throwing an exception if we keep spinning
looking for a victim page.
Does that make sense to you?
Sattam
Original comment by salsuba...@gmail.com
on 22 Jun 2013 at 9:09
Original comment by salsuba...@gmail.com
on 26 Jun 2013 at 5:40
Original issue reported on code.google.com by
pouria.p...@gmail.com
on 19 Jun 2013 at 12:13Attachments: