mk-log-player produces huge number of files

[GoogleCodeExporter]

If this is a problem report, what steps will reproduce the problem?

This issue may come up when slow log is really big. In my environment it
was 68G.
1. Run "mk-log-player --split Thread_id slow_log"

What is the expected output? What do you see instead?

There are several problems may appear.
1. The max files per directory limit can be reached.
2. any_command mysql-split* will fail with error meaage "Too many arguments"
3. ls, du and other application will be extreamly slow.

What information do you get from mk-<toolname>--version?

# mk-log-player --version
mk-log-player  Ver @VERSION@ Distrib @DISTRIB@ Changeset 2371

What is your MySQL, Perl, DBI, and DBD::mysql version?

N/A

What is your operating system/distribution?

Linux sm1u02 2.6.18-53.el5 #1 SMP Mon Nov 12 02:14:55 EST 2007 x86_64
x86_64 x86_64 GNU/Linux

There are two solutions are seen:
1. To save files in a directory hierarchy based on hash function.
2. Instead of splitting slow log - build an index and use it to play the
slow log.

Original issue reported on code.google.com by aleksand...@gmail.com on 4 Dec 2008 at 9:59

[GoogleCodeExporter]

Original comment by dan...@percona.com on 12 Dec 2008 at 4:53

• Added labels: Priority-High, Tool-mk_log_player
• Removed labels: Priority-Medium

[GoogleCodeExporter]

I think solution 1 in combination with a few new options will work well:
--maxsessions: stop after parsing this many sessions
--maxfiles: max files per directory (up to --maxsessions)
--maxdirs: max directories to create, each holding --maxfiles (up to 
--maxsessions)

What do you think?

Original comment by dan...@percona.com on 12 Dec 2008 at 5:14

• Changed title: mk-log-player produces huge number of files
• Changed state: Accepted

[GoogleCodeExporter]

This has been implemented. Any other feature requests?

Original comment by dan...@percona.com on 15 Dec 2008 at 10:15

• Changed title: mk-log-player feature requests
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I have found that this is not the best place to take feature requests.  We need 
to
start an email thread and decide on new design specs especially given all the 
changes
we've done recently.

Summary was "mk-log-player feature requests".

Original comment by dan...@percona.com on 5 Jun 2009 at 3:11

• Changed title: mk-log-player produces huge number of files
• Changed state: Fixed