absl flags/tf.app.flags

[AndrewLischishin]

Hi there! I've tried for a while to run your code, but still have some flag problems, so here what I get:

this is what I get if I just 'make' in nips2016:

jupyter nbconvert --inplace --execute --ExecutePreprocessor.timeout=-1 20news.ipynb
[NbConvertApp] Converting notebook 20news.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] ERROR | Error while converting '20news.ipynb'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 393, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 174, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 192, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/exporters/notebook.py", line 31, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 134, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 311, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 262, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 286, in preprocess_cell
    raise CellExecutionError.from_cell_and_msg(cell, out)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:

------------------

# Fetch dataset. Scikit-learn already performs some cleaning.
remove = ('headers','footers','quotes')  # (), ('headers') or ('headers','footers','quotes')
train = utils.Text20News(data_home=FLAGS.dir_data, subset='train', remove=remove)

#Pre-processing: transform everything to a-z and whitespace.
print(train.show_document(1)[:400])
train.clean_text(num='substitute')

# Analyzing / tokenizing: transform documents to bags-of-words.
#stop_words = set(sklearn.feature_extraction.text.ENGLISH_STOP_WORDS)
# Or stop words from NLTK.
# Add e.g. don, ve.
train.vectorize(stop_words='english')
print(train.show_document(1)[:400])

------------------

------------------------------------------------------------------------

UnrecognizedFlagError                     Traceback (most recent call last)
<ipython-input-3-0b079301697c> in <module>()
      1 # Fetch dataset. Scikit-learn already performs some cleaning.
      2 remove = ('headers','footers','quotes')  # (), ('headers') or ('headers','footers','quotes')
----> 3 train = utils.Text20News(data_home=FLAGS.dir_data, subset='train', remove=remove)
      4 
      5 # Pre-processing: transform everything to a-z and whitespace.

/usr/local/lib/python3.6/site-packages/tensorflow/python/platform/flags.py in __getattr__(self, name)
     82     # a flag.
     83     if not wrapped.is_parsed():
---> 84       wrapped(_sys.argv)
     85     return wrapped.__getattr__(name)
     86 

/usr/local/lib/python3.6/site-packages/absl/flags/_flagvalues.py in __call__(self, argv, known_only)
    628       suggestions = _helpers.get_flag_suggestions(name, list(self))
    629       raise _exceptions.UnrecognizedFlagError(
--> 630           name, value, suggestions=suggestions)
    631 
    632     self.mark_as_parsed()

UnrecognizedFlagError: Unknown command line flag 'f'
UnrecognizedFlagError: Unknown command line flag 'f' 

I tried to solve it myself and what I found was, that tensorflow uses now absl for flags, so I tried to do this:

from absl import flags
from absl import logging
from absl import app
FLAGS = flags.FLAGS

instead of

#flags = tf.app.flags
#FLAGS = flags.FLAGS

but still get another error

# Graphs.
flags.DEFINE_integer('number_edges', 8, 'Graph: minimum number of edges per vertex.')
flags.DEFINE_string('metric', 'euclidean', 'Graph: similarity measure (between features).')
# TODO: change cgcnn for combinatorial Laplacians.
flags.DEFINE_bool('normalized_laplacian', True, 'Graph Laplacian: normalized.')
flags.DEFINE_integer('coarsening_levels', 4, 'Number of coarsened graphs.')

#Directories.
flags.DEFINE_string('dir_data', os.path.join('..', 'data', 'mnist'), 'Directory to store data.')`

   def grid_graph(m, corners=False):
    z = graph.grid(m)
    dist, idx = graph.distance_sklearn_metrics(z, k=FLAGS.number_edges, metric=FLAGS.metric)
    A = graph.adjacency(dist, idx)

    # Connections are only vertical or horizontal on the grid.
    # Corner vertices are connected to 2 neightbors only.
    if corners:
        import scipy.sparse
        A = A.toarray()
        A[A < A.max()/1.5] = 0
        A = scipy.sparse.csr_matrix(A)
        print('{} edges'.format(A.nnz))

    print("{} > {} edges".format(A.nnz//2, FLAGS.number_edges*m**2//2))
    return A

    t_start = time.process_time()
    A = grid_graph(28, corners=False)  
    A = graph.replace_random_edges(A, 0)
    graphs, perm = coarsening.coarsen(A, levels=FLAGS.coarsening_levels, self_connections=False)
    L = [graph.laplacian(A, normalized=True) for A in graphs]
    print('Execution time: {:.2f}s'.format(time.process_time() - t_start))
   graph.plot_spectrum(L)
   del A

---------------------------------------------------------------------------

UnparsedFlagAccessError                   Traceback (most recent call last)
<ipython-input-3-b674f783dcfd> in <module>()
     17 
     18 t_start = time.process_time()
---> 19 A = grid_graph(28, corners=False)
     20 A = graph.replace_random_edges(A, 0)
     21 graphs, perm = coarsening.coarsen(A, levels=FLAGS.coarsening_levels, self_connections=False)

<ipython-input-3-b674f783dcfd> in grid_graph(m, corners)
      1 def grid_graph(m, corners=False):
      2     z = graph.grid(m)
----> 3     dist, idx = graph.distance_sklearn_metrics(z, k=FLAGS.number_edges, metric=FLAGS.metric)
      4     A = graph.adjacency(dist, idx)
      5 

/usr/local/lib/python3.6/site-packages/absl/flags/_flagvalues.py in __getattr__(self, name)
    486         # get too much noise.
    487         logging.error(error_message)
--> 488       raise _exceptions.UnparsedFlagAccessError(error_message)
    489 
    490   def __setattr__(self, name, value):

UnparsedFlagAccessError: Trying to access flag --number_edges before flags were parsed.

So, maybe I just overlook something, but still stuck in it.

[hurun]

i also stuck this problem,and not solve it still now.

[AndrewLischishin]

So, this problem is also discussed here, and it pops out because of using jupyter notebook. In order to be able to run mnist.py(or any other) just convert the .ipynb file into .py, for example with this:

$ jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

then you can just normally execute with:

python<version> file.py 

if some matplotlib problems pop out check this, maybe it will answer your questions, or this. I am using MacBook Pro 2014mid, macOS High Sierra, Version 10.13.3, python 3.6.4, tensorflow 1.6.0. and I didn't instal tensorflow-gpu from requirements.txt(brought some errors in). The first solution worked for me, when the second one just brought some blocking while executing. If someone found other, more simple, solution or maybe it is just correct as it is, please comment.

[prateeky2806]

I am facing the same issue, after converting to .py file there are lots of errors to resolve. Isn't there a way to fix this in jupyter notebook only?

[TMorville]

To anyone having this issue in the notebook, add tf.app.flags.DEFINE_string('f', '', 'kernel') before defining your flags.

E.g. for rcv1

flags = tf.app.flags
FLAGS = flags.FLAGS

#
tf.app.flags.DEFINE_string('f', '', 'kernel')
# Graphs.
flags.DEFINE_integer('number_edges', 16, 'Graph: minimum number of edges per vertex.')
flags.DEFINE_string('metric', 'cosine', 'Graph: similarity measure (between features).')
# TODO: change cgcnn for combinatorial Laplacians.
flags.DEFINE_bool('normalized_laplacian', True, 'Graph Laplacian: normalized.')
flags.DEFINE_integer('coarsening_levels', 0, 'Number of coarsened graphs.')

flags.DEFINE_string('dir_data', os.path.join('data', 'rcv1'), 'Directory to store data.')
flags.DEFINE_integer('val_size', 400, 'Size of the validation set.')

[mdeff]

That's probably due to a TensorFlow update. The original code should work with tensorflow-gpu==1.1.0 (the version I used to develop the code).

If the fix (adding tf.app.flags.DEFINE_string('f', '', 'kernel')) also works with tensorflow-gpu==1.1.0 I'd accept a PR.