search

weecology / NEON_crown_maps

Generating tree crown maps for NEON sites
MIT License
1 stars 0 forks source link

Missing pngs #29

Closed bw4sz closed 4 years ago

bw4sz commented 4 years ago

I am trying to solve quite a weird mystery.

Writing to a folder that has lots of files fails

(base) [b.weinstein@login3 NEON_crown_maps]$ ls /orange/idtrees-collab/crops/tfrecords/ | wc -l
6308805
>>> create_tfrecords('/orange/ewhite/NeonData/SRER/DP3.30010.001/2019/FullSite/D14/2019_SRER_3/L3/Camera/Mosaic/2019_SRER_3_521000_3521000_image.tif', patch_size=400, patch_overlap=0.05, savedir="/orange/idtrees-collab/crops/tfrecords/",overwrite=False)
There are 729 windows
'/orange/idtrees-collab/crops/tfrecords/2019_SRER_3_521000_3521000_image.tfrecord'
>>> run_rgb(['/orange/idtrees-collab/crops/tfrecords/2019_SRER_3_521000_3521000_image.tfrecord'], ['/orange/ewhite/NeonData/SRER/DP3.30010.001/2019/FullSite/D14/2019_SRER_3/L3/Camera/Mosaic/2019_SRER_3_521000_3521000_image.tif'])
Reading config file: /home/b.weinstein/miniconda3/envs/crowns/lib/python3.7/site-packages/deepforest/data/deepforest_config.yml
Creating model from weights
tracking <tf.Variable 'Variable_10:0' shape=(9, 4) dtype=float32> anchors
tracking <tf.Variable 'Variable_11:0' shape=(9, 4) dtype=float32> anchors
tracking <tf.Variable 'Variable_12:0' shape=(9, 4) dtype=float32> anchors
tracking <tf.Variable 'Variable_13:0' shape=(9, 4) dtype=float32> anchors
tracking <tf.Variable 'Variable_14:0' shape=(9, 4) dtype=float32> anchors
2020-04-04 17:31:53.624878: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /orange/idtrees-collab/crops/tfrecords/2019_SRER_3_521000_3521000_image_495.png; No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/b.weinstein/NEON_crown_maps/main.py", line 144, in run_rgb
    shps = predict.predict_tiles(model, records, patch_size=400, rgb_paths=rgb_paths, save_dir=save_dir, batch_size=model.config["batch_size"],overwrite=overwrite)
  File "/home/b.weinstein/NEON_crown_maps/predict.py", line 67, in predict_tiles
    boxes = predict_tile(model=model, tfrecord=tfrecord, patch_size=patch_size, batch_size=batch_size, score_threshold=score_threshold, max_detections=max_detections, classes=classes)
  File "/home/b.weinstein/NEON_crown_maps/predict.py", line 140, in predict_tile
    box_array, score_array, label_array = model.prediction_model.predict_on_batch(iterator)
  File "/apps/tensorflow/1.14.0.cuda10.gpu/lib/python3.7/site-packages/keras/engine/training.py", line 1580, in predict_on_batch
    outputs = self.predict_function(ins)
  File "/apps/tensorflow/1.14.0.cuda10.gpu/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/apps/tensorflow/1.14.0.cuda10.gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) Not found: /orange/idtrees-collab/crops/tfrecords/2019_SRER_3_521000_3521000_image_495.png; No such file or directory
     [[{{node ReadFile}}]]
     [[IteratorGetNext_2]]
     [[filtered_detections_2/map/while/non_max_suppression/NonMaxSuppressionV3/_5703]]
  (1) Not found: /orange/idtrees-collab/crops/tfrecords/2019_SRER_3_521000_3521000_image_495.png; No such file or directory
     [[{{node ReadFile}}]]
     [[IteratorGetNext_2]]
0 successful operations.
0 derived errors ignored.

Writing the same file to the directory next to it succeeds

create_tfrecords('/orange/ewhite/NeonData/SRER/DP3.30010.001/2019/FullSite/D14/2019_SRER_3/L3/Camera/Mosaic/2019_SRER_3_521000_3521000_image.tif', patch_size=400, patch_overlap=0.05, savedir="/orange/idtrees-collab/test/",overwrite=False)

run_rgb(['/orange/idtrees-collab/test/2019_SRER_3_521000_3521000_image.tfrecord'], ['/orange/ewhite/NeonData/SRER/DP3.30010.001/2019/FullSite/D14/2019_SRER_3/L3/Camera/Mosaic/2019_SRER_3_521000_3521000_image.tif'])

Locally it succeeds

def test_yields_all_pngs():
    """On hipergator there is a rare error where some .png are skipped, using a sample tile that erred. Due to large file size this can only be run locally as debug"""
    tfrecords.create_tfrecords('/Users/ben/Downloads/hipergator/2019_SRER_3_521000_3521000_image.tif', patch_size=400, patch_overlap=0.05, savedir="output",overwrite=True)
    for i in np.arange(729):
        assert os.path.exists("output/2019_SRER_3_521000_3521000_image_{}.png".format(i))