NPT-14408: Unable to see data when importing tensorboard logs.

odinsbane commented 9 months ago

I am not able to view metrics from tensorboard generated logs.

1. To reproduce

Train a tensorflow model with keras using the tesnorboad callback.

import keras
import numpy

inp = keras.layers.Input( (16, ) )
op = keras.layers.Dense(4)(inp)

mdl = keras.models.Model( inputs=[inp], outputs = [op] )

x = numpy.random.random((128, 16))
y = (numpy.random.random((128, 4))>0.75) * 1.0

mdl.compile()

metrics = [ keras.metrics.BinaryAccuracy(name = "ba")]
mdl.compile( optimizer = keras.optimizers.Adam(0.00001), loss="mean_squared_error", metrics = metrics)

callbacks = [ keras.callbacks.TensorBoard(log_dir='./junk-logs'), keras.callbacks.EarlyStopping(patience=5, monitor="loss", mode='min')]

mdl.fit( x, y, callbacks=callbacks, epochs=100)

Then I try to synch the logs with the neptune.

export NEPTUNE_PROJECT="project/name"
export NEPTUNE_API_TOKEN="xxx"
./neptune-env/bin/neptune tensorboard junk-logs

The result is a new entry in my project, but I cannot find any of the values that were tracked.

Expected behavior

I can see graphs for the metrics in tensorboard, but I cannot see the graphs in neptune

3. Environment

Default environment that was installed when I used ./neptune-env/bin/pip install neptune-tensorboard

./neptune-env/bin/python --version

> Python 3.10.13

./neptune-env/bin/pip list

Package Version

absl-py 2.0.0 arrow 1.3.0 astunparse 1.6.3 attrs 23.2.0 boto3 1.34.14 botocore 1.34.14 bravado 11.0.3 bravado-core 6.1.1 cachetools 5.3.2 certifi 2023.11.17 charset-normalizer 3.3.2 click 8.1.7 contourpy 1.2.0 cycler 0.12.1 exceptiongroup 1.2.0 execnet 2.0.2 filelock 3.13.1 flatbuffers 23.5.26 fonttools 4.47.0 fqdn 1.5.1 fsspec 2023.12.2 future 0.18.3 gast 0.5.4 gitdb 4.0.11 GitPython 3.1.40 google-auth 2.26.1 google-auth-oauthlib 1.2.0 google-pasta 0.2.0 grpcio 1.60.0 h5py 3.10.0 idna 3.6 iniconfig 2.0.0 isoduration 20.11.0 Jinja2 3.1.2 jmespath 1.0.1 jsonpointer 2.4 jsonref 1.1.0 jsonschema 4.20.0 jsonschema-specifications 2023.12.1 keras 2.15.0 kiwisolver 1.4.5 libclang 16.0.6 Markdown 3.5.1 MarkupSafe 2.1.3 matplotlib 3.8.2 ml-dtypes 0.2.0 monotonic 1.6 mpmath 1.3.0 msgpack 1.0.7 neptune 1.8.6 neptune-tensorboard 1.0.2 networkx 3.2.1 numpy 1.26.3 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.18.1 nvidia-nvjitlink-cu12 12.3.101 nvidia-nvtx-cu12 12.1.105 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 23.2 pandas 2.1.4 pillow 10.2.0 pip 23.0.1 pluggy 1.3.0 protobuf 4.23.4 psutil 5.9.7 pyasn1 0.5.1 pyasn1-modules 0.3.0 PyJWT 2.8.0 pyparsing 3.1.1 pytest 7.4.4 pytest-xdist 3.5.0 python-dateutil 2.8.2 pytz 2023.3.post1 PyYAML 6.0.1 referencing 0.32.1 requests 2.31.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.16.2 rsa 4.9 s3transfer 0.10.0 setuptools 65.5.0 simplejson 3.19.2 six 1.16.0 smmap 5.0.1 swagger-spec-validator 3.0.3 sympy 1.12 tbparse 0.0.8 tensorboard 2.15.1 tensorboard-data-server 0.7.2 tensorboardX 2.6.2.2 tensorflow 2.15.0.post1 tensorflow-estimator 2.15.0 tensorflow-io-gcs-filesystem 0.35.0 termcolor 2.4.0 tomli 2.0.1 torch 2.1.2 triton 2.1.0 types-python-dateutil 2.8.19.14 typing_extensions 4.9.0 tzdata 2023.4 uri-template 1.3.0 urllib3 2.0.7 webcolors 1.13 websocket-client 1.7.0 Werkzeug 3.0.1 wheel 0.41.2 wrapt 1.14.1

This is a 64bit linux, I think centos with 4.18 kernel.

SiddhantSadangi commented 9 months ago

Thanks for reporting this @odinsbane 🙇

I was able to reproduce this and have created a ticket for the engineering team. I'll keep you updated on the progress and reach out if we need any further information.

AleksanderWWW commented 8 months ago

@SiddhantSadangi @odinsbane I've identified the issue to be a gap in what we log in the integration automatically (the field tensors was missing and actually it was the only field to which any data was logged in this example). I believe that https://github.com/neptune-ai/neptune-tensorboard/pull/71 will sort things out

neptune-ai / neptune-tensorboard