LINQ to Text Files

[GoogleCodeExporter]

LINQ to Text Files
EricWhite
31 Aug 2006 6:51 PM

    * Comments 8

(Sept 30, 2008 - I've changed my approach for querying text files.  The new 
approach is detailed in LINQ to TEXT and LINQ to CSV.) 

Lazy evaluation is an important technique in functional programming. There is a 
entertaining article on functional programming here.

I was thinking about lazy evaluation the other day, and the issues of 
processing huge text files using streaming techniques, and I realized that LINQ 
could do really cool things if I implemented an extension method for 
StreamReader that iterated over the lines of a text file. Because of the lazy 
evaluation, provided you construct only certain queries, this technique doesn't 
read the entire file into memory. Each line is read, processed by both LINQ 
query expressions, output to the console, and then the next line is read. 
However, be aware that certain LINQ operators such as orderby can force the 
entire file to be read into memory.

The following example is somewhat artificial; it could be implemented with a 
single query expression, but I wanted to use two query expressions to 
demonstrate the laziness of processing. No lines are read from the text file 
until the program iterates over the results of the second query expression.

I just gotta say, this is the expressiveness that I REALLY wanted many years 
ago when I was writing Unix shell scripts to manipulate text files.

The implementation is trivial. The following listing contains both the 
extension method as well as the code to use the extension method:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Linq;

namespace LinqToText
{
  public static class StreamReaderSequence
  {
    public static IEnumerable<string> Lines(this StreamReader source)
    {
      String line;

      if (source == null)
          throw new ArgumentNullException("source");
      while ((line = source.ReadLine()) != null)
      {
        yield return line;
      }
    }
  }

  class Program
  {
    static void Main(string[] args)
    {
      StreamReader sr = new StreamReader("TextFile.txt");

      var t1 =
        from line in sr.Lines()
        let items = line.Split(',')
        where ! line.StartsWith("#")
        select String.Format("{0}{1}{2}",
            items[1].PadRight(16),
            items[2].PadRight(16),
            items[3].PadRight(16));

      var t2 =
        from line in t1
        select line.ToUpper();

      foreach (var t in t2)
        Console.WriteLine(t);

      sr.Close();
    }
  }
}

If you run this example with the following text file:

#This is a comment
1,Eric,White,Writer
2,Bob,Jones,Programmer
3,Orville,Wright,Inventor
4,Thomas,Jefferson,Statesman
5,George,Washington,President

It produces the following output.

ERIC            WHITE           WRITER
BOB             JONES           PROGRAMMER
ORVILLE         WRIGHT          INVENTOR
THOMAS          JEFFERSON       STATESMAN
GEORGE          WASHINGTON      PRESIDENT

Original issue reported on code.google.com by cn.wei.hp@gmail.com on 15 Dec 2010 at 7:46

[GoogleCodeExporter]

Account_ID: 98734BLAH9873
User Name: something_85
First Name: ILove
Last Name: XML
Age: 209

var pairs = File.ReadAllLines("filename.txt")
    .Select(line => line.Split(':'))
    .ToDictionary(cells => cells[0].Trim(), cells => cells[1].Trim())

Original comment by cn.wei.hp@gmail.com on 15 Dec 2010 at 8:02