search

crowsonkb / style_transfer

Data-parallel image stylization using Caffe.
MIT License
113 stars 14 forks source link

bug: exploding gradient? #18

Open moofin2017 opened 7 years ago

moofin2017 commented 7 years ago

Seem to have run into a bug where the loss exploded. See below, step 337:

Using Ubuntu 16.04, Python3.6, CUDNN6.1, CUDA8.0, MKL, GPU.

Parameters: aux_image: None aux_weight: 10 avg_window: 20 caffe_path: None config: None content_image: 'XXX' content_layers: ['conv4_2'] content_weight: 0.05 dd_layers: [] dd_weight: 0 debug: False devices: [0] display: 'browser' div: 1 hidpi: False init_image: 'XXX' iterations: [500] jitter: False layer_weights: None list_layers: False mean: (103.939, 116.779, 123.68) min_size: 640 model: 'vgg19.prototxt' optimizer: 'lbfgs' output_image: '/home/XXX/tmp.jpg' p_power: 0.0 p_weight: 0.0 port: 8000 prompt: False save_every: 50 seed: 123 size: 640 step_decay: [0.05, 0.5] step_size: 15 style_images: ['XXX] style_layers: ['conv1_2', 'conv2_2', 'conv3_2', 'conv4_2'] style_scale: 1 style_scale_up: False swt_levels: 1 swt_power: 0.0 swt_wavelet: 'haar' swt_weight: 0 tile_size: 640 tv_power: 2 tv_weight: 5 weights: 'vgg19.caffemodel'

Run 170609_213442 started. MKL detected, 2 threads maximum. Initializing vgg19.caffemodel.

Watch the progress at: http://127.0.0.1:8000/

Starting 1 worker process(es).

Scale 1, image size 636x640.

Preprocessing the style image(s)...

Exception happened during processing of request from ('XXX', 26477) Traceback (most recent call last): File "/home/XXX/miniconda3/lib/python3.6/socketserver.py", line 639, in process_request_thread self.finish_request(request, client_address) File "/home/XXX/miniconda3/lib/python3.6/socketserver.py", line 361, in finish_request self.RequestHandlerClass(request, client_address, self) File "/home/XXX/miniconda3/lib/python3.6/socketserver.py", line 696, in init self.handle() File "/home/XXX/miniconda3/lib/python3.6/http/server.py", line 418, in handle self.handle_one_request() File "/home/XXX/miniconda3/lib/python3.6/http/server.py", line 406, in handle_one_request method() File "style_transfer.py", line 937, in do_GET 'w': self.server.transfer.current_output.size[0] / scale, AttributeError: 'NoneType' object has no attribute 'size'

Preprocessing the content image(s)... Step 1, time: 0.00 s, update: 0.10, loss: 10419.5, tv: 107.1 Step 2, time: 1.24 s, update: 0.08, loss: 10608.3, tv: 107.2 Step 3, time: 1.21 s, update: 0.14, loss: 10827.2, tv: 107.7 Step 4, time: 1.21 s, update: 0.27, loss: 12920.2, tv: 108.9 Step 5, time: 1.21 s, update: 0.47, loss: 13270, tv: 112.3 Step 6, time: 1.22 s, update: 0.76, loss: 15437.3, tv: 121.0 Step 7, time: 1.23 s, update: 1.50, loss: 19740.8, tv: 148.7 Step 8, time: 1.26 s, update: 2.11, loss: 24616.9, tv: 208.8 Step 9, time: 1.26 s, update: 2.89, loss: 28553.7, tv: 318.5 Step 10, time: 1.28 s, update: 5.04, loss: 38511.2, tv: 587.0 Step 11, time: 1.27 s, update: 1.15, loss: 31685.8, tv: 550.0 Step 12, time: 1.26 s, update: 0.84, loss: 30629.5, tv: 501.0 Step 13, time: 1.27 s, update: 2.82, loss: 32331.3, tv: 499.6 Step 14, time: 1.26 s, update: 6.86, loss: 36716.1, tv: 673.9 Step 15, time: 1.26 s, update: 2.80, loss: 37888.8, tv: 703.0 Step 16, time: 1.26 s, update: 2.51, loss: 35133.7, tv: 653.0 Step 17, time: 1.26 s, update: 1.36, loss: 36051.4, tv: 701.4 Step 18, time: 1.26 s, update: 4.59, loss: 41514.2, tv: 869.6 Step 19, time: 1.27 s, update: 5.32, loss: 40490.7, tv: 993.6 Step 20, time: 1.26 s, update: 2.87, loss: 44028.8, tv: 1046.9 Step 21, time: 1.28 s, update: 4.95, loss: 41657.2, tv: 1268.4 Step 22, time: 1.27 s, update: 4.61, loss: 44222.9, tv: 1467.1 Step 23, time: 1.26 s, update: 3.77, loss: 47761.4, tv: 1459.9 Step 24, time: 1.26 s, update: 1.44, loss: 45711.8, tv: 1418.0 Step 25, time: 1.26 s, update: 6.80, loss: 50723.4, tv: 1789.5 Step 26, time: 1.26 s, update: 2.30, loss: 44947.8, tv: 1658.8 Step 27, time: 1.27 s, update: 2.28, loss: 43336, tv: 1489.8 Step 28, time: 1.27 s, update: 2.56, loss: 45194.7, tv: 1517.2 Step 29, time: 1.26 s, update: 4.51, loss: 48040.3, tv: 1651.6 Step 30, time: 1.25 s, update: 0.88, loss: 45854, tv: 1603.2 Step 31, time: 1.25 s, update: 1.04, loss: 43793.9, tv: 1554.6 Step 32, time: 1.25 s, update: 3.13, loss: 43440.1, tv: 1479.2 Step 33, time: 1.27 s, update: 0.70, loss: 47759, tv: 1489.8 Step 34, time: 1.27 s, update: 1.67, loss: 46123.7, tv: 1558.6 Step 35, time: 1.28 s, update: 0.85, loss: 48269.7, tv: 1526.4 Step 36, time: 1.29 s, update: 1.76, loss: 47131.9, tv: 1458.7 Step 37, time: 1.27 s, update: 1.22, loss: 43511.4, tv: 1497.3 Step 38, time: 1.26 s, update: 1.27, loss: 49965.3, tv: 1521.7 Step 39, time: 1.28 s, update: 1.26, loss: 46585.2, tv: 1538.2 Step 40, time: 1.27 s, update: 1.27, loss: 49179.4, tv: 1547.3 Step 41, time: 1.27 s, update: 0.68, loss: 44348.5, tv: 1521.4 Step 42, time: 1.26 s, update: 0.53, loss: 43117.8, tv: 1517.6 Step 43, time: 1.27 s, update: 4.36, loss: 50655.6, tv: 1537.8 Step 44, time: 1.26 s, update: 2.50, loss: 47224.7, tv: 1477.1 Step 45, time: 1.27 s, update: 0.98, loss: 43886.3, tv: 1525.6 Step 46, time: 1.27 s, update: 0.84, loss: 45925.4, tv: 1514.0 Step 47, time: 1.26 s, update: 1.91, loss: 44424.7, tv: 1515.9 Step 48, time: 1.26 s, update: 1.13, loss: 46069.3, tv: 1522.6 Step 49, time: 1.26 s, update: 0.75, loss: 47045.3, tv: 1519.3 Step 50, time: 1.26 s, update: 1.79, loss: 45196.8, tv: 1550.1 Step 51, time: 1.42 s, update: 0.84, loss: 45395.4, tv: 1548.5 Step 52, time: 1.26 s, update: 1.11, loss: 47724.1, tv: 1546.0 Step 53, time: 1.26 s, update: 1.39, loss: 43643.8, tv: 1535.4 Step 54, time: 1.26 s, update: 0.87, loss: 47452.7, tv: 1551.1 Step 55, time: 1.27 s, update: 0.79, loss: 46669.7, tv: 1540.8 Step 56, time: 1.27 s, update: 0.61, loss: 43584.7, tv: 1546.9 Step 57, time: 1.27 s, update: 0.61, loss: 46596.4, tv: 1541.8 Step 58, time: 1.26 s, update: 0.34, loss: 44365, tv: 1539.0 Step 59, time: 1.26 s, update: 1.05, loss: 44153.4, tv: 1544.5 Step 60, time: 1.27 s, update: 1.05, loss: 48819.3, tv: 1545.2 Step 61, time: 1.28 s, update: 0.49, loss: 47519, tv: 1543.8 Step 62, time: 1.28 s, update: 0.35, loss: 43438.5, tv: 1548.1 Step 63, time: 1.28 s, update: 0.63, loss: 46905.3, tv: 1559.8 Step 64, time: 1.28 s, update: 0.70, loss: 44537.1, tv: 1573.8 Step 65, time: 1.27 s, update: 0.27, loss: 43964.9, tv: 1568.1 Step 66, time: 1.27 s, update: 0.36, loss: 43557.8, tv: 1557.8 Step 67, time: 1.27 s, update: 0.86, loss: 42951.9, tv: 1560.3 Step 68, time: 1.27 s, update: 1.99, loss: 48516.2, tv: 1580.3 Step 69, time: 1.30 s, update: 0.79, loss: 43484.6, tv: 1567.5 Step 70, time: 1.27 s, update: 0.20, loss: 43782.3, tv: 1566.8 Step 71, time: 1.31 s, update: 1.47, loss: 48041.1, tv: 1579.3 Step 72, time: 1.28 s, update: 0.37, loss: 46904.4, tv: 1567.5 Step 73, time: 1.27 s, update: 0.37, loss: 42385.6, tv: 1561.8 Step 74, time: 1.28 s, update: 0.36, loss: 43642.9, tv: 1564.6 Step 75, time: 1.27 s, update: 2.00, loss: 42926.7, tv: 1592.4 Step 76, time: 1.27 s, update: 1.04, loss: 45708.9, tv: 1571.2 Step 77, time: 1.27 s, update: 0.29, loss: 44430, tv: 1568.6 Step 78, time: 1.26 s, update: 0.39, loss: 42952, tv: 1571.3 Step 79, time: 1.27 s, update: 1.28, loss: 42604.8, tv: 1587.1 Step 80, time: 1.29 s, update: 0.26, loss: 41702.3, tv: 1580.1 Step 81, time: 1.26 s, update: 1.07, loss: 43014.1, tv: 1577.9 Step 82, time: 1.27 s, update: 0.60, loss: 43239.8, tv: 1580.8 Step 83, time: 1.26 s, update: 0.63, loss: 42986.6, tv: 1581.3 Step 84, time: 1.28 s, update: 0.87, loss: 42529.5, tv: 1588.6 Step 85, time: 1.27 s, update: 0.75, loss: 43069.9, tv: 1591.0 Step 86, time: 1.29 s, update: 0.96, loss: 42920.8, tv: 1595.2 Step 87, time: 1.26 s, update: 1.14, loss: 46280.5, tv: 1600.8 Step 88, time: 1.28 s, update: 0.76, loss: 46194.6, tv: 1581.3 Step 89, time: 1.26 s, update: 0.68, loss: 46422.2, tv: 1601.7 Step 90, time: 1.26 s, update: 0.55, loss: 41958.6, tv: 1611.0 Step 91, time: 1.30 s, update: 0.26, loss: 42871.3, tv: 1603.0 Step 92, time: 1.27 s, update: 0.57, loss: 44943.7, tv: 1596.5 Step 93, time: 1.26 s, update: 0.20, loss: 42430.5, tv: 1600.9 Step 94, time: 1.28 s, update: 0.18, loss: 47354.5, tv: 1602.2 Step 95, time: 1.26 s, update: 0.29, loss: 47591.7, tv: 1601.3 Step 96, time: 1.29 s, update: 0.40, loss: 45342.7, tv: 1591.5 Step 97, time: 1.26 s, update: 0.29, loss: 43592, tv: 1594.8 Step 98, time: 1.27 s, update: 0.40, loss: 47920.6, tv: 1604.3 Step 99, time: 1.27 s, update: 0.55, loss: 47027.1, tv: 1613.1 Step 100, time: 1.26 s, update: 0.36, loss: 42541.4, tv: 1612.9 Step 101, time: 1.44 s, update: 0.12, loss: 42383.2, tv: 1607.2 Step 102, time: 1.27 s, update: 0.24, loss: 43558.7, tv: 1595.6 Step 103, time: 1.27 s, update: 0.44, loss: 42899.1, tv: 1603.8 Step 104, time: 1.27 s, update: 0.87, loss: 46694.6, tv: 1618.9 Step 105, time: 1.27 s, update: 0.18, loss: 42755.9, tv: 1613.4 Step 106, time: 1.26 s, update: 0.14, loss: 43713.4, tv: 1608.7 Step 107, time: 1.30 s, update: 0.43, loss: 42285.9, tv: 1607.3 Step 108, time: 1.28 s, update: 0.49, loss: 41530, tv: 1609.6 Step 109, time: 1.28 s, update: 1.12, loss: 46926.8, tv: 1612.5 Step 110, time: 1.28 s, update: 0.76, loss: 47044.5, tv: 1603.1 Step 111, time: 1.28 s, update: 0.88, loss: 42571.2, tv: 1605.1 Step 112, time: 1.27 s, update: 0.54, loss: 43078.1, tv: 1603.0 Step 113, time: 1.28 s, update: 1.72, loss: 48049.8, tv: 1615.4 Step 114, time: 1.31 s, update: 1.13, loss: 40836.5, tv: 1603.0 Step 115, time: 1.26 s, update: 0.23, loss: 43116.4, tv: 1609.7 Step 116, time: 1.28 s, update: 0.15, loss: 43369.8, tv: 1608.7 Step 117, time: 1.27 s, update: 0.70, loss: 43694.8, tv: 1604.6 Step 118, time: 1.26 s, update: 0.72, loss: 47495.1, tv: 1597.8 Step 119, time: 1.28 s, update: 0.49, loss: 42729.5, tv: 1597.8 Step 120, time: 1.27 s, update: 0.14, loss: 43332.4, tv: 1598.9 Step 121, time: 1.43 s, update: 0.63, loss: 45208.1, tv: 1604.8 Step 122, time: 1.44 s, update: 0.38, loss: 45420, tv: 1611.8 Step 123, time: 1.28 s, update: 0.29, loss: 44619.9, tv: 1610.0 Step 124, time: 1.27 s, update: 0.12, loss: 45856.9, tv: 1604.0 Step 125, time: 1.27 s, update: 0.31, loss: 43856.8, tv: 1595.8 Step 126, time: 1.28 s, update: 0.53, loss: 43147.8, tv: 1597.1 Step 127, time: 1.28 s, update: 1.01, loss: 41329.4, tv: 1604.2 Step 128, time: 1.29 s, update: 0.16, loss: 45157.3, tv: 1598.4 Step 129, time: 1.27 s, update: 0.32, loss: 42119.2, tv: 1593.3 Step 130, time: 1.27 s, update: 0.18, loss: 48178.5, tv: 1592.9 Step 131, time: 1.26 s, update: 0.28, loss: 42228.6, tv: 1597.1 Step 132, time: 1.27 s, update: 0.98, loss: 47369.9, tv: 1618.1 Step 133, time: 1.27 s, update: 0.73, loss: 43483.1, tv: 1603.8 Step 134, time: 1.28 s, update: 0.27, loss: 46274, tv: 1593.8 Step 135, time: 1.27 s, update: 0.08, loss: 43756.4, tv: 1594.7 Step 136, time: 1.29 s, update: 0.27, loss: 42082, tv: 1597.9 Step 137, time: 1.28 s, update: 0.29, loss: 45029.3, tv: 1598.6 Step 138, time: 1.27 s, update: 0.46, loss: 43465.6, tv: 1602.7 Step 139, time: 1.28 s, update: 0.08, loss: 44160.6, tv: 1600.1 Step 140, time: 1.27 s, update: 0.44, loss: 43267.5, tv: 1591.0 Step 141, time: 1.27 s, update: 0.35, loss: 48134.3, tv: 1598.0 Step 142, time: 1.32 s, update: 0.53, loss: 45536.6, tv: 1612.6 Step 143, time: 1.28 s, update: 0.26, loss: 46811, tv: 1603.9 Step 144, time: 1.28 s, update: 0.62, loss: 45713.4, tv: 1588.7 Step 145, time: 1.29 s, update: 0.44, loss: 42027.9, tv: 1597.5 Step 146, time: 1.31 s, update: 0.33, loss: 41982.8, tv: 1603.1 Step 147, time: 1.27 s, update: 0.33, loss: 42575.2, tv: 1606.2 Step 148, time: 1.27 s, update: 0.28, loss: 45002.6, tv: 1605.2 Step 149, time: 1.27 s, update: 0.10, loss: 45937.4, tv: 1602.2 Step 150, time: 1.27 s, update: 0.12, loss: 46353.6, tv: 1596.5 Step 151, time: 1.47 s, update: 0.38, loss: 46685.7, tv: 1588.2 Step 152, time: 1.27 s, update: 0.19, loss: 45457.3, tv: 1592.9 Step 153, time: 1.28 s, update: 0.86, loss: 43034.3, tv: 1614.3 Step 154, time: 1.27 s, update: 0.35, loss: 42644.1, tv: 1602.3 Step 155, time: 1.27 s, update: 0.08, loss: 43023.7, tv: 1603.0 Step 156, time: 1.33 s, update: 0.21, loss: 43476.5, tv: 1603.0 Step 157, time: 1.28 s, update: 0.16, loss: 41774.2, tv: 1602.5 Step 158, time: 1.27 s, update: 0.41, loss: 42673.3, tv: 1603.4 Step 159, time: 1.27 s, update: 0.17, loss: 42295.5, tv: 1599.9 Step 160, time: 1.27 s, update: 0.26, loss: 41017.6, tv: 1599.2 Step 161, time: 1.31 s, update: 1.51, loss: 44959.7, tv: 1614.8 Step 162, time: 1.28 s, update: 0.77, loss: 49049.2, tv: 1605.0 Step 163, time: 1.28 s, update: 42.13, loss: 168899, tv: 5795.0 Step 164, time: 1.27 s, update: 37.43, loss: 49161.9, tv: 1692.2 Step 165, time: 1.28 s, update: 39.12, loss: 150652, tv: 4698.0 Step 166, time: 1.28 s, update: 26.49, loss: 46724.5, tv: 1836.4 Step 167, time: 1.27 s, update: 26.73, loss: 53644, tv: 2572.9 Step 168, time: 1.26 s, update: 13.89, loss: 51063.5, tv: 1797.5 Step 169, time: 1.27 s, update: 7.62, loss: 40470.5, tv: 1637.1 Step 170, time: 1.26 s, update: 3.25, loss: 44657.2, tv: 1635.1 Step 171, time: 1.28 s, update: 1.99, loss: 53084.3, tv: 1583.5 Step 172, time: 1.31 s, update: 0.84, loss: 46487.9, tv: 1555.1 Step 173, time: 1.28 s, update: 1.66, loss: 45507.8, tv: 1573.9 Step 174, time: 1.27 s, update: 1.33, loss: 43401.3, tv: 1533.8 Step 175, time: 1.29 s, update: 0.94, loss: 49324.4, tv: 1541.9 Step 176, time: 1.28 s, update: 0.75, loss: 43716, tv: 1567.0 Step 177, time: 1.27 s, update: 0.58, loss: 44135, tv: 1583.2 Step 178, time: 1.26 s, update: 0.39, loss: 48379.2, tv: 1583.0 Step 179, time: 1.26 s, update: 0.74, loss: 45292.7, tv: 1577.2 Step 180, time: 1.26 s, update: 0.58, loss: 42067.1, tv: 1589.9 Step 181, time: 1.26 s, update: 0.89, loss: 45413.2, tv: 1609.4 Step 182, time: 1.29 s, update: 0.37, loss: 44313.5, tv: 1599.6 Step 183, time: 1.27 s, update: 0.27, loss: 44068.3, tv: 1592.3 Step 184, time: 1.28 s, update: 0.46, loss: 43475, tv: 1593.7 Step 185, time: 1.28 s, update: 0.92, loss: 44620.3, tv: 1604.7 Step 186, time: 1.28 s, update: 0.21, loss: 49654.9, tv: 1598.3 Step 187, time: 1.31 s, update: 1.41, loss: 44526.8, tv: 1561.7 Step 188, time: 1.28 s, update: 0.85, loss: 43978.8, tv: 1583.8 Step 189, time: 1.27 s, update: 2.02, loss: 44019.7, tv: 1648.5 Step 190, time: 1.27 s, update: 1.77, loss: 43256.5, tv: 1601.5 Step 191, time: 1.26 s, update: 0.55, loss: 42632, tv: 1594.5 Step 192, time: 1.29 s, update: 0.15, loss: 43002.6, tv: 1597.7 Step 193, time: 1.28 s, update: 0.24, loss: 43531.4, tv: 1597.6 Step 194, time: 1.28 s, update: 0.18, loss: 42603.9, tv: 1596.2 Step 195, time: 1.26 s, update: 0.57, loss: 43905.6, tv: 1593.1 Step 196, time: 1.27 s, update: 0.13, loss: 45307.4, tv: 1591.3 Step 197, time: 1.30 s, update: 0.50, loss: 46003.8, tv: 1587.4 Step 198, time: 1.31 s, update: 0.15, loss: 42384.5, tv: 1595.5 Step 199, time: 1.28 s, update: 0.58, loss: 43278.7, tv: 1605.0 Step 200, time: 1.45 s, update: 0.09, loss: 44077.1, tv: 1600.4 Step 201, time: 1.44 s, update: 0.17, loss: 44015, tv: 1590.5 Step 202, time: 1.27 s, update: 0.16, loss: 48677.8, tv: 1589.8 Step 203, time: 1.26 s, update: 0.45, loss: 47429.4, tv: 1588.7 Step 204, time: 1.26 s, update: 0.29, loss: 40954.4, tv: 1595.8 Step 205, time: 1.27 s, update: 0.16, loss: 42134.9, tv: 1594.3 Step 206, time: 1.27 s, update: 0.20, loss: 46770.6, tv: 1593.2 Step 207, time: 1.29 s, update: 0.10, loss: 48480.7, tv: 1591.5 Step 208, time: 1.26 s, update: 1.09, loss: 43384.4, tv: 1592.3 Step 209, time: 1.29 s, update: 0.59, loss: 45010.4, tv: 1586.0 Step 210, time: 1.28 s, update: 0.61, loss: 42447.6, tv: 1602.1 Step 211, time: 1.26 s, update: 0.29, loss: 44121.4, tv: 1593.9 Step 212, time: 1.28 s, update: 0.45, loss: 47598.4, tv: 1587.4 Step 213, time: 1.26 s, update: 0.18, loss: 42110.6, tv: 1590.9 Step 214, time: 1.26 s, update: 0.43, loss: 45882.6, tv: 1598.7 Step 215, time: 1.26 s, update: 0.25, loss: 41043.1, tv: 1593.6 Step 216, time: 1.27 s, update: 0.05, loss: 42887.8, tv: 1592.7 Step 217, time: 1.27 s, update: 0.07, loss: 44725.3, tv: 1592.5 Step 218, time: 1.32 s, update: 0.22, loss: 43801.2, tv: 1590.9 Step 219, time: 1.28 s, update: 0.10, loss: 47323.8, tv: 1592.1 Step 220, time: 1.28 s, update: 0.57, loss: 41405.1, tv: 1597.6 Step 221, time: 1.28 s, update: 0.45, loss: 41370.4, tv: 1593.1 Step 222, time: 1.28 s, update: 0.49, loss: 49801, tv: 1589.1 Step 223, time: 1.27 s, update: 0.26, loss: 45823.6, tv: 1590.3 Step 224, time: 1.26 s, update: 0.19, loss: 49813.7, tv: 1592.6 Step 225, time: 1.26 s, update: 0.89, loss: 43216.2, tv: 1605.1 Step 226, time: 1.27 s, update: 0.74, loss: 45954.5, tv: 1594.1 Step 227, time: 1.26 s, update: 0.67, loss: 42847.8, tv: 1585.7 Step 228, time: 1.29 s, update: 0.23, loss: 46812.4, tv: 1589.0 Step 229, time: 1.26 s, update: 0.36, loss: 47178.1, tv: 1595.4 Step 230, time: 1.26 s, update: 0.13, loss: 41816.8, tv: 1593.3 Step 231, time: 1.27 s, update: 1.04, loss: 40021.5, tv: 1580.7 Step 232, time: 1.26 s, update: 0.61, loss: 41633.1, tv: 1586.6 Step 233, time: 1.31 s, update: 0.13, loss: 42902.1, tv: 1590.9 Step 234, time: 1.28 s, update: 0.12, loss: 43484.9, tv: 1590.3 Step 235, time: 1.26 s, update: 0.08, loss: 46059.7, tv: 1589.5 Step 236, time: 1.27 s, update: 0.04, loss: 44352, tv: 1588.5 Step 237, time: 1.27 s, update: 0.06, loss: 43593.6, tv: 1588.1 Step 238, time: 1.30 s, update: 0.21, loss: 46775.3, tv: 1585.7 Step 239, time: 1.27 s, update: 0.03, loss: 42130.4, tv: 1586.0 Step 240, time: 1.27 s, update: 0.03, loss: 42924.7, tv: 1586.9 Step 241, time: 1.28 s, update: 0.11, loss: 46523.1, tv: 1587.5 Step 242, time: 1.28 s, update: 0.11, loss: 43822.8, tv: 1587.3 Step 243, time: 1.27 s, update: 0.21, loss: 47894.8, tv: 1588.4 Step 244, time: 1.28 s, update: 0.09, loss: 43719, tv: 1588.4 Step 245, time: 1.27 s, update: 0.06, loss: 43897.7, tv: 1587.8 Step 246, time: 1.27 s, update: 1.16, loss: 41907.3, tv: 1584.2 Step 247, time: 1.27 s, update: 1.06, loss: 48081.9, tv: 1588.2 Step 248, time: 1.31 s, update: 0.33, loss: 45782.8, tv: 1590.8 Step 249, time: 1.28 s, update: 0.30, loss: 46932.1, tv: 1588.3 Step 250, time: 1.28 s, update: 0.45, loss: 42192.1, tv: 1587.0 Step 251, time: 1.46 s, update: 0.34, loss: 42037.4, tv: 1587.8 Step 252, time: 1.28 s, update: 0.57, loss: 46370.6, tv: 1589.4 Step 253, time: 1.31 s, update: 0.25, loss: 45724, tv: 1588.1 Step 254, time: 1.27 s, update: 0.32, loss: 43830.7, tv: 1589.0 Step 255, time: 1.26 s, update: 0.25, loss: 48036.4, tv: 1589.4 Step 256, time: 1.27 s, update: 0.29, loss: 43246.9, tv: 1591.3 Step 257, time: 1.27 s, update: 0.05, loss: 50157.4, tv: 1590.8 Step 258, time: 1.28 s, update: 0.07, loss: 43477.1, tv: 1589.9 Step 259, time: 1.27 s, update: 0.13, loss: 41631.5, tv: 1590.6 Step 260, time: 1.27 s, update: 0.08, loss: 40849.9, tv: 1590.3 Step 261, time: 1.27 s, update: 3.85, loss: 43073.7, tv: 1630.1 Step 262, time: 1.28 s, update: 3.41, loss: 46868.2, tv: 1569.3 Step 263, time: 1.29 s, update: 1.63, loss: 43590.7, tv: 1620.2 Step 264, time: 1.27 s, update: 1.19, loss: 45749.1, tv: 1591.5 Step 265, time: 1.28 s, update: 0.58, loss: 41652.3, tv: 1579.7 Step 266, time: 1.27 s, update: 0.12, loss: 46153.6, tv: 1582.7 Step 267, time: 1.28 s, update: 0.32, loss: 41510.4, tv: 1589.2 Step 268, time: 1.27 s, update: 0.22, loss: 41011.7, tv: 1585.6 Step 269, time: 1.28 s, update: 0.45, loss: 45227.5, tv: 1579.9 Step 270, time: 1.27 s, update: 0.22, loss: 43783.8, tv: 1583.1 Step 271, time: 1.27 s, update: 0.82, loss: 42861.4, tv: 1598.4 Step 272, time: 1.27 s, update: 0.64, loss: 43179, tv: 1586.3 Step 273, time: 1.26 s, update: 0.27, loss: 42369.2, tv: 1581.7 Step 274, time: 1.30 s, update: 0.07, loss: 41997.9, tv: 1582.9 Step 275, time: 1.29 s, update: 0.08, loss: 42190, tv: 1583.6 Step 276, time: 1.28 s, update: 0.18, loss: 44237.5, tv: 1583.7 Step 277, time: 1.26 s, update: 0.05, loss: 42257.9, tv: 1583.3 Step 278, time: 1.28 s, update: 0.21, loss: 45037.6, tv: 1584.5 Step 279, time: 1.30 s, update: 0.06, loss: 43770.9, tv: 1584.6 Step 280, time: 1.27 s, update: 0.08, loss: 42658.5, tv: 1584.0 Step 281, time: 1.26 s, update: 0.10, loss: 42048, tv: 1583.4 Step 282, time: 1.26 s, update: 0.60, loss: 41648.7, tv: 1583.1 Step 283, time: 1.28 s, update: 0.89, loss: 47764.4, tv: 1586.4 Step 284, time: 1.31 s, update: 0.64, loss: 47340, tv: 1583.8 Step 285, time: 1.28 s, update: 1.23, loss: 42114.6, tv: 1588.5 Step 286, time: 1.28 s, update: 0.62, loss: 43256.2, tv: 1587.3 Step 287, time: 1.26 s, update: 0.62, loss: 42762.7, tv: 1584.8 Step 288, time: 1.26 s, update: 0.65, loss: 42238.5, tv: 1586.4 Step 289, time: 1.26 s, update: 0.05, loss: 42354.9, tv: 1585.8 Step 290, time: 1.27 s, update: 0.25, loss: 45837.1, tv: 1586.2 Step 291, time: 1.26 s, update: 0.71, loss: 42247.6, tv: 1584.1 Step 292, time: 1.27 s, update: 0.42, loss: 43747.2, tv: 1585.7 Step 293, time: 1.29 s, update: 0.26, loss: 43386.9, tv: 1588.4 Step 294, time: 1.32 s, update: 0.15, loss: 45547.5, tv: 1587.4 Step 295, time: 1.27 s, update: 0.44, loss: 43684.7, tv: 1584.6 Step 296, time: 1.28 s, update: 0.22, loss: 41008.4, tv: 1586.1 Step 297, time: 1.27 s, update: 1.03, loss: 43187.3, tv: 1595.5 Step 298, time: 1.26 s, update: 0.61, loss: 41148.6, tv: 1588.4 Step 299, time: 1.27 s, update: 0.15, loss: 42319.4, tv: 1589.0 Step 300, time: 1.27 s, update: 0.20, loss: 46695.1, tv: 1590.2 Step 301, time: 1.44 s, update: 0.09, loss: 43520.7, tv: 1589.4 Step 302, time: 1.27 s, update: 0.03, loss: 47327.5, tv: 1589.6 Step 303, time: 1.26 s, update: 0.21, loss: 43698.3, tv: 1588.1 Step 304, time: 1.26 s, update: 0.08, loss: 42767, tv: 1587.9 Step 305, time: 1.27 s, update: 0.15, loss: 42011.9, tv: 1588.5 Step 306, time: 1.25 s, update: 0.25, loss: 44237.7, tv: 1590.4 Step 307, time: 1.26 s, update: 0.18, loss: 41857.7, tv: 1591.2 Step 308, time: 1.26 s, update: 0.71, loss: 48018.6, tv: 1597.2 Step 309, time: 1.28 s, update: 0.43, loss: 42145.5, tv: 1592.7 Step 310, time: 1.30 s, update: 0.06, loss: 49154, tv: 1592.9 Step 311, time: 1.29 s, update: 0.05, loss: 47452.3, tv: 1592.0 Step 312, time: 1.27 s, update: 0.17, loss: 41987.4, tv: 1589.2 Step 313, time: 1.27 s, update: 0.10, loss: 46838.4, tv: 1589.8 Step 314, time: 1.27 s, update: 0.13, loss: 43961.2, tv: 1590.3 Step 315, time: 1.30 s, update: 0.05, loss: 44114.1, tv: 1590.2 Step 316, time: 1.27 s, update: 0.25, loss: 43453.7, tv: 1590.3 Step 317, time: 1.28 s, update: 0.81, loss: 41966.2, tv: 1595.0 Step 318, time: 1.28 s, update: 0.27, loss: 44221.1, tv: 1593.5 Step 319, time: 1.28 s, update: 0.30, loss: 42698.4, tv: 1589.9 Step 320, time: 1.30 s, update: 0.07, loss: 44151, tv: 1590.5 Step 321, time: 1.26 s, update: 0.37, loss: 44571.6, tv: 1594.2 Step 322, time: 1.26 s, update: 0.05, loss: 48659.5, tv: 1594.2 Step 323, time: 1.27 s, update: 0.51, loss: 47120.7, tv: 1589.0 Step 324, time: 1.26 s, update: 0.47, loss: 43107.7, tv: 1594.7 Step 325, time: 1.28 s, update: 0.24, loss: 41114.3, tv: 1597.5 Step 326, time: 1.28 s, update: 0.32, loss: 43694, tv: 1591.3 Step 327, time: 1.26 s, update: 0.27, loss: 44330.4, tv: 1595.3 Step 328, time: 1.27 s, update: 0.75, loss: 44922.4, tv: 1607.1 Step 329, time: 1.26 s, update: 0.53, loss: 45132.3, tv: 1597.6 Step 330, time: 1.29 s, update: 0.22, loss: 42533.3, tv: 1594.1 Step 331, time: 1.29 s, update: 0.09, loss: 44107.6, tv: 1594.8 Step 332, time: 1.28 s, update: 0.37, loss: 44405.4, tv: 1599.9 Step 333, time: 1.27 s, update: 0.14, loss: 41805, tv: 1597.6 Step 334, time: 1.28 s, update: 0.08, loss: 43102.9, tv: 1595.7 Step 335, time: 1.28 s, update: 0.10, loss: 42902.5, tv: 1594.8 Step 336, time: 1.27 s, update: 0.12, loss: 45430.3, tv: 1596.4 Step 337, time: 1.26 s, update: 124.05, loss: 4.66159e+06, tv: 53820.1 Step 338, time: 1.27 s, update: 128.86, loss: 175108, tv: 5941.8 Step 339, time: 1.27 s, update: 414.61, loss: 3.7417e+08, tv: 572740.2 Step 340, time: 1.26 s, update: 308.64, loss: 4.64603e+06, tv: 46223.9 Step 341, time: 1.30 s, update: 452.16, loss: 3.17311e+08, tv: 511010.3 Step 342, time: 1.27 s, update: 284.55, loss: 2.01989e+06, tv: 50430.3 Step 343, time: 1.27 s, update: 365.84, loss: 9.2684e+07, tv: 291202.2 Step 344, time: 1.27 s, update: 228.72, loss: 856938, tv: 27716.4 Step 345, time: 1.27 s, update: 299.16, loss: 6.70247e+07, tv: 211315.2 Step 346, time: 1.34 s, update: 184.42, loss: 522272, tv: 22322.3 Step 347, time: 1.28 s, update: 297.39, loss: 5.39132e+07, tv: 200266.1 Step 348, time: 1.26 s, update: 183.17, loss: 456517, tv: 18500.5 Step 349, time: 1.27 s, update: 409.75, loss: 2.92089e+08, tv: 397985.7 Step 350, time: 1.28 s, update: 260.85, loss: 2.06018e+06, tv: 32128.0 Step 351, time: 1.48 s, update: 614.65, loss: 1.74065e+09, tv: 871743.9 Step 352, time: 1.28 s, update: 383.60, loss: 1.16935e+07, tv: 64054.6 Step 353, time: 1.29 s, update: 870.44, loss: 8.37409e+09, tv: 1535512.6 Step 354, time: 1.27 s, update: 552.77, loss: 3.3467e+07, tv: 95403.3

crowsonkb commented 6 years ago

It looks like L-BFGS took a bad step and was unable to recover. Unfortunately my L-BFGS implementation does not include a line search to guard against and reject bad steps. I very rarely saw them in practice. The only answer I can really give with the current implementation is to use Adam instead for these inputs.