SU-ECE-17-7 / hotspotter

Apache License 2.0
0 stars 1 forks source link

Query Problems #10

Closed AudreyBeard closed 7 years ago

AudreyBeard commented 7 years ago

Affected Branches:

All Branches Affected

To Reproduce:

  1. Open HS
  2. Import images (if not already done)
  3. Define chips using one of the following methods:
    • Manually
      • Navigate/click: Menu Bar->Actions->Add Chip
    • Autochipping
      • Click AutoChip
  4. Navigate to the Chip Table
  5. Select a chip
  6. Using the menu bar: Actions -> Query

    Output:

    
    [**back] query(cid=None)
    [**back.query()] cx = 0)

==================== [hs] query database

[mc3] FEAT_UID is different. Need to reload features [mc3] Old: None [mc3] New: _FEAT(hesaff+sift,0_9001)_CHIP(sz750) [hs] unload_cxdata(cx='all') [hs] Unloading all data [hs] finished unloading all data

/home/joshuabeard/code/hotspotter/hotspotter/HotSpotterAPI.py(345)refresh_features() -> hs.load_chips(cx_list=cx_list) (Pdb) n

============================= [cc2] Precomputing chips and loading chip paths: 'test0'

[cc2] chip_uid = '_CHIP(sz750)' [parallel] Already computed 1 compute_chip tasks

tic('Distributing 8 compute_chip: tasks to 8 processes') compute_chip: compute_chip: 8/8 ...toc('Distributing 8 compute_chip: tasks to 8 processes')=0.2567s [xcb] Unknown sequence number while processing queue [xcb] Most likely this is a multi-threaded client and XInitThreads has not been called [xcb] Aborting, sorry about that. python: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed. [cc2] Done Precomputing chips and loading chip paths [cc2]=============================

/home/joshuabeard/code/hotspotter/hotspotter/HotSpotterAPI.py(346)refresh_features() -> hs.load_features(cx_list=cx_list) (Pdb) n

============================= [fc2] Precomputing and loading features: 'test0'

[fc2] feat_uid = '_FEAT(hesaff+sift,0_9001)_CHIP(sz750)' [io] fname='kpts_list_FEAT(hesaff+sift,0_9001)_CHIP(sz750)_cids((9,)5l$qyq1{).npy' does not exist [io] fname='desc_list_FEAT(hesaff+sift,0_9001)_CHIP(sz750)_cids((9,)5l$qyq1{).npy' does not exist [fc2] Loading _FEAT(hesaff+sift,0_9001)_CHIP(sz750) individually [parallel] Already computed 0 precompute_hesaff tasks

tic('Distributing 9 precompute_hesaff: tasks to 8 processes') precompute_hesaff: Error in `python': malloc(): smallbin double linked list corrupted: 0x0000000005057510 ======= Backtrace: =========

The next bajillion lines are memory map garbage, with the following message thrown in:

Error in `python': free(): corrupted unsorted chunks: 0x0000000005057dd0


Followed by another bajillion lines of (apparent) garbage
AudreyBeard commented 7 years ago

Original note:

It seems that chip_uid is set to the exact same (default, '_CHIP(sz750)') for each chip. My guess is that this issue happens at the point of populating the chip table. it seems that 'sz' is short for 'size' and that '750' is the default value for the square root of the area.

Response:

After exploring this further, it seems that if we add a chip the old-fashioned way (using Actions->Add Chip), the chip_uid has the same default name. Because of this, I don't think this is the issue.

probably Deprecated Work:

Following chip_compute2.py chip_cfg = hs.prefs.chip_cfg chip_uid = chip_cfg.get_uid()

Leads to HotSpotterAPI.py hs.prefs.chip_cfg = Config.default_chip_cfg()

To Config.py

def default_chip_cfg(**kwargs):
    chip_cfg = ChipConfig(**kwargs)
    return chip_cfg

AND

class ChipConfig(ConfigBase):
    def __init__(cc_cfg, **kwargs):
        super(ChipConfig, cc_cfg).__init__(name='chip_cfg')
        cc_cfg.chip_sqrt_area = 750
        cc_cfg.grabcut         = False
        cc_cfg.histeq          = False
        cc_cfg.adapteq         = False
        cc_cfg.region_norm     = False
        cc_cfg.rank_eq         = False
        cc_cfg.local_eq        = False
        cc_cfg.maxcontrast     = False
        cc_cfg.update(**kwargs)

AND ConfigBase = Pref AND from hscom.Preferences import Pref

AudreyBeard commented 7 years ago

This issue exists even if we produce a chip the old-fashioned way.

AudreyBeard commented 7 years ago

Backtracing (reverse chronologically):

  1. ERROR
  2. hscom/Parallelize.py
    • Who calls this?
  3. feature_compute2.load_features
    • Calls hscom/fileio.__smart_load (L 170) in a weird roundabout way (maybe through hs.prefs.feat_cfg
    • fpath does not exist
      • I want to find out where kpts_list_FEAT(hesaff+sift,0_9001)_CHIP(sz750)_cids((9,)5l$qyq1{).npy comes from. Where is that file written?
  4. hotspotter/feature_compute2._load_features_individually (feature_compute2.py L 101)
    • called by feature_compute2.load_features
AudreyBeard commented 7 years ago

Broken part:

parallel_compute(precompute_fn, precompute_args, **pfc_kwargs) Which lives in feature_compute2._load_features_individualy

mattdioso commented 7 years ago

I did kind of a work around to make queries compute in serial rather than parallel: within the if statement, I made it compute in serial for both conditions. In hscom/Parallelize.py:

def parallelize_tasks(task_list, num_procs, task_lbl='', verbose=True):
    '''
    Used for embarissingly parallel tasks, which write output to disk
    '''
    nTasks = len(task_list)
    msg = ('Distributing %d %s tasks to %d processes' % (nTasks, task_lbl, num_procs)
           if num_procs > 1 else
           'Executing %d %s tasks in serial' % (nTasks, task_lbl))
    with helpers.Timer(msg=msg):
        if num_procs >= 1:
            # Parallelize tasks
            #return _compute_in_parallel(task_list, num_procs, task_lbl, verbose)
            return _compute_in_serial(task_list, task_lbl, verbose)
        else:
            return _compute_in_serial(task_list, task_lbl, verbose)
AudreyBeard commented 7 years ago

^ This is a hacky patch. I'm comfortable with this fix for the sake of a demo, but we cannot close this issue yet, as the underlying problem isn't fixed.

AudreyBeard commented 7 years ago

Following hscom/Parallelize._compute_in_serial():

pdb trace:

> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(118)detect_kpts()
-> hesaff_lib.exportArrays(hesaff_ptr, nKpts, kpts, desc)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(196)from_param()
-> @classmethod
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(198)from_param()
-> if not isinstance(obj, ndarray):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(200)from_param()
-> if cls._dtype_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(201)from_param()
-> and obj.dtype != cls._dtype_:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(203)from_param()
-> if cls._ndim_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(204)from_param()
-> and obj.ndim != cls._ndim_:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(206)from_param()
-> if cls._shape_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(209)from_param()
-> if cls._flags_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(210)from_param()
-> and ((obj.flags.num & cls._flags_) != cls._flags_):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()
-> return obj.ctypes
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(225)__init__()
-> def __init__(self, array, ptr=None):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(226)__init__()
-> try:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(227)__init__()
-> self._ctypes = ctypes
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(230)__init__()
-> self._arr = array
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(231)__init__()
-> self._data = ptr
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(232)__init__()
-> if self._arr.ndim == 0:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(235)__init__()
-> self._zerod = False
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(235)__init__()->None
-> self._zerod = False
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()-><numpy.c...fe9e1750>
-> return obj.ctypes
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(263)get_as_parameter()
-> def get_as_parameter(self):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()
-> return self._ctypes.c_void_p(self._data)
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()->c_void_p(59365568)
-> return self._ctypes.c_void_p(self._data)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(196)from_param()
-> @classmethod
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(198)from_param()
-> if not isinstance(obj, ndarray):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(200)from_param()
-> if cls._dtype_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(201)from_param()
-> and obj.dtype != cls._dtype_:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(203)from_param()
-> if cls._ndim_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(204)from_param()
-> and obj.ndim != cls._ndim_:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(206)from_param()
-> if cls._shape_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(209)from_param()
-> if cls._flags_ is not None \
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(210)from_param()
-> and ((obj.flags.num & cls._flags_) != cls._flags_):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()
-> return obj.ctypes
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(225)__init__()
-> def __init__(self, array, ptr=None):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(226)__init__()
-> try:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(227)__init__()
-> self._ctypes = ctypes
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(230)__init__()
-> self._arr = array
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(231)__init__()
-> self._data = ptr
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(232)__init__()
-> if self._arr.ndim == 0:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(235)__init__()
-> self._zerod = False
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(235)__init__()->None
-> self._zerod = False
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()-><numpy.c...fe9e1750>
-> return obj.ctypes
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(263)get_as_parameter()
-> def get_as_parameter(self):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()
-> return self._ctypes.c_void_p(self._data)
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()->c_void_p(54037840)
-> return self._ctypes.c_void_p(self._data)
(Pdb) s
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(120)detect_kpts()
-> if use_adaptive_scale:
(Pdb) s
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(123)detect_kpts()
-> return kpts, desc
(Pdb) s
--Return--
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(123)detect_kpts()->(array([[...=float32), array([[...pe=uint8))
-> return kpts, desc
(Pdb) s
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(59)detect_kpts_new()
-> return kpts, desc
(Pdb) s
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(59)detect_kpts_new()->(array([[...=float32), array([[...pe=uint8))
-> return kpts, desc
(Pdb) s
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(89)compute_hesaff()->(array([[...=float32), array([[...pe=uint8))
-> return detect_kpts(rchip_fpath, dict_args)
(Pdb) rchip_fpath
'/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid1_CHIP(sz750).png'
(Pdb) s
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(43)precompute()
-> np.savez(feat_fpath, kpts, desc)
(Pdb) feat_fpath
'/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid1_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz'
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(515)savez()
-> def savez(file, *args, **kwds):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(593)savez()
-> _savez(file, args, kwds, False)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(622)_savez()
-> def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(625)_savez()
-> import zipfile
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(627)_savez()
-> import tempfile
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(629)_savez()
-> if isinstance(file, basestring):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(630)_savez()
-> if not file.endswith('.npz'):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(636)_savez()
-> namedict = kwds
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(637)_savez()
-> for i, val in enumerate(args):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(638)_savez()
-> key = 'arr_%d' % i
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(639)_savez()
-> if key in namedict.keys():
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(642)_savez()
-> namedict[key] = val
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(637)_savez()
-> for i, val in enumerate(args):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(638)_savez()
-> key = 'arr_%d' % i
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(639)_savez()
-> if key in namedict.keys():
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(642)_savez()
-> namedict[key] = val
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(637)_savez()
-> for i, val in enumerate(args):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(644)_savez()
-> if compress:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(647)_savez()
-> compression = zipfile.ZIP_STORED
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(649)_savez()
-> zipf = zipfile_factory(file, mode="w", compression=compression)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(89)zipfile_factory()
-> def zipfile_factory(file, *args, **kwargs):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(97)zipfile_factory()
-> if is_pathlib_path(file):
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/compat/py3k.py(94)is_pathlib_path()
-> def is_pathlib_path(obj):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/compat/py3k.py(98)is_pathlib_path()
-> return Path is not None and isinstance(obj, Path)
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/compat/py3k.py(98)is_pathlib_path()->False
-> return Path is not None and isinstance(obj, Path)
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(99)zipfile_factory()
-> import zipfile
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(100)zipfile_factory()
-> kwargs['allowZip64'] = True
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(101)zipfile_factory()
-> return zipfile.ZipFile(file, *args, **kwargs)
(Pdb) s
--Call--
> /usr/lib/python2.7/zipfile.py(726)__init__()
-> def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=False):
(Pdb) s
> /usr/lib/python2.7/zipfile.py(728)__init__()
-> if mode not in ("r", "w", "a"):
(Pdb) s
> /usr/lib/python2.7/zipfile.py(731)__init__()
-> if compression == ZIP_STORED:
(Pdb) s
> /usr/lib/python2.7/zipfile.py(732)__init__()
-> pass
(Pdb) s
> /usr/lib/python2.7/zipfile.py(740)__init__()
-> self._allowZip64 = allowZip64
(Pdb) s
> /usr/lib/python2.7/zipfile.py(741)__init__()
-> self._didModify = False
(Pdb) s
> /usr/lib/python2.7/zipfile.py(742)__init__()
-> self.debug = 0  # Level of printing: 0 through 3
(Pdb) s
> /usr/lib/python2.7/zipfile.py(743)__init__()
-> self.NameToInfo = {}    # Find file info given name
(Pdb) s
> /usr/lib/python2.7/zipfile.py(744)__init__()
-> self.filelist = []      # List of ZipInfo instances for archive
(Pdb) s
> /usr/lib/python2.7/zipfile.py(745)__init__()
-> self.compression = compression  # Method of compression
(Pdb) s
> /usr/lib/python2.7/zipfile.py(746)__init__()
-> self.mode = key = mode.replace('b', '')[0]
(Pdb) s
> /usr/lib/python2.7/zipfile.py(747)__init__()
-> self.pwd = None
(Pdb) s
> /usr/lib/python2.7/zipfile.py(748)__init__()
-> self._comment = ''
(Pdb) s
> /usr/lib/python2.7/zipfile.py(751)__init__()
-> if isinstance(file, basestring):
(Pdb) s
> /usr/lib/python2.7/zipfile.py(752)__init__()
-> self._filePassed = 0
(Pdb) s
> /usr/lib/python2.7/zipfile.py(753)__init__()
-> self.filename = file
(Pdb) s
> /usr/lib/python2.7/zipfile.py(754)__init__()
-> modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'}
(Pdb) s
> /usr/lib/python2.7/zipfile.py(755)__init__()
-> try:
(Pdb) s
> /usr/lib/python2.7/zipfile.py(756)__init__()
-> self.fp = open(file, modeDict[mode])
(Pdb) mode
'w'
(Pdb) s
> /usr/lib/python2.7/zipfile.py(768)__init__()
-> try:
(Pdb) s
> /usr/lib/python2.7/zipfile.py(769)__init__()
-> if key == 'r':
(Pdb) s
> /usr/lib/python2.7/zipfile.py(771)__init__()
-> elif key == 'w':
(Pdb) s
> /usr/lib/python2.7/zipfile.py(774)__init__()
-> self._didModify = True
(Pdb) s
--Return--
> /usr/lib/python2.7/zipfile.py(774)__init__()->None
-> self._didModify = True
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(101)zipfile_factory()-><zipfile...fe8521d0>
-> return zipfile.ZipFile(file, *args, **kwargs)
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(655)_savez()
-> file_dir, file_prefix = os.path.split(file) if _is_string_like(file) else (None, 'tmp')
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/_iotools.py(31)_is_string_like()
-> def _is_string_like(obj):
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/_iotools.py(35)_is_string_like()
-> try:
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/_iotools.py(36)_is_string_like()
-> obj + ''
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/_iotools.py(39)_is_string_like()
-> return True
(Pdb) s
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/_iotools.py(39)_is_string_like()->True
-> return True
(Pdb) s
--Call--
> /usr/lib/python2.7/posixpath.py(82)split()
-> def split(p):
(Pdb) s
> /usr/lib/python2.7/posixpath.py(85)split()
-> i = p.rfind('/') + 1
(Pdb) p
*** SyntaxError: SyntaxError('unexpected EOF while parsing', ('<string>', 0, 0, ''))
(Pdb) s
> /usr/lib/python2.7/posixpath.py(86)split()
-> head, tail = p[:i], p[i:]
(Pdb) s
> /usr/lib/python2.7/posixpath.py(87)split()
-> if head and head != '/'*len(head):
(Pdb) p[0]
[0]
(Pdb) p[1]
[1]
(Pdb) p[2]
[2]
(Pdb) s
> /usr/lib/python2.7/posixpath.py(88)split()
-> head = head.rstrip('/')
(Pdb) s
> /usr/lib/python2.7/posixpath.py(89)split()
-> return head, tail
(Pdb) s
--Return--
> /usr/lib/python2.7/posixpath.py(89)split()->('/home/joshua...omputed/feats', 'cid1_FEAT(he...IP(sz750).npz')
-> return head, tail
(Pdb) s
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/lib/npyio.py(656)_savez()
-> fd, tmpfile = tempfile.mkstemp(prefix=file_prefix, dir=file_dir, suffix='-numpy.npy')
(Pdb) s
--Call--
> /usr/lib/python2.7/tempfile.py(280)mkstemp()
-> def mkstemp(suffix="", prefix=template, dir=None, text=False):
(Pdb) s
> /usr/lib/python2.7/tempfile.py(306)mkstemp()
-> if dir is None:
(Pdb) s
> /usr/lib/python2.7/tempfile.py(309)mkstemp()
-> if text:
(Pdb) s
> /usr/lib/python2.7/tempfile.py(312)mkstemp()
-> flags = _bin_openflags
(Pdb) s
> /usr/lib/python2.7/tempfile.py(314)mkstemp()
-> return _mkstemp_inner(dir, prefix, suffix, flags)
(Pdb) s
--Call--
> /usr/lib/python2.7/tempfile.py(235)_mkstemp_inner()
-> def _mkstemp_inner(dir, pre, suf, flags):
(Pdb) s
Segmentation fault (core dumped)
AudreyBeard commented 7 years ago

I'm seeing _hsdb/computed/feats/cid1_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npzQIMND8-numpy.npy when we have a file called _hsdb/computed/feats/cid1_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz

IDK if this is an issue, ultimately, but it is weird.

AudreyBeard commented 7 years ago

Same problem, different situation:

precompute_hesaff: 1/9> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(118)detect_kpts()
-> hesaff_lib.exportArrays(hesaff_ptr, nKpts, kpts, desc)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(196)from_param()
-> @classmethod
(Pdb) r
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()-><numpy.c...3bdb8b10>
-> return obj.ctypes
(Pdb) r
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(120)detect_kpts()
-> if use_adaptive_scale:
(Pdb) a
img_fpath = /home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid1_CHIP(sz750).png
use_adaptive_scale = False
kwargs = {'scale_min': 0, 'scale_max': 9001}
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(123)detect_kpts()
-> return kpts, desc
(Pdb) kpts
array([[ 211.2383728 ,   23.7825222 ,   17.11903191,   -4.12547588,
           6.25793791],
       [   0.        ,  173.53675842,   34.55820084,    7.89015245,
          -3.02890778],
       [  16.8110733 ,    0.        ,  509.90692139,   41.80162811,
          17.89713097],
       ..., 
       [ 266.46685791,   59.65991592,    8.03850842,   36.32440948,    0.        ],
       [  93.73831177,  276.43423462,   42.18911362,   -1.61140704,
          60.05384064],
       [   0.        ,  451.71258545,  276.43289185,   37.56335831,
          -6.43680286]], dtype=float32)
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(123)detect_kpts()->(array([[...=float32), array([[...pe=uint8))
-> return kpts, desc
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(59)detect_kpts_new()
-> return kpts, desc
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(59)detect_kpts_new()->(array([[...=float32), array([[...pe=uint8))
-> return kpts, desc
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(89)compute_hesaff()->(array([[...=float32), array([[...pe=uint8))
-> return detect_kpts(rchip_fpath, dict_args)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(43)precompute()
-> np.savez(feat_fpath, kpts, desc)
(Pdb) feat_fpath
'/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid1_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz'
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(44)precompute()
-> return kpts, desc
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(44)precompute()->(array([[...=float32), array([[...pe=uint8))
-> return kpts, desc
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(48)precompute_hesaff()->(array([[...=float32), array([[...pe=uint8))
-> return precompute(rchip_fpath, feat_fpath, dict_args, compute_hesaff)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hscom/Parallelize.py(118)_compute_in_serial()
-> result_list.append(result)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hscom/Parallelize.py(114)_compute_in_serial()
-> for count, (fn, args) in enumerate(task_list):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hscom/Parallelize.py(115)_compute_in_serial()
-> mark_progress(count)
(Pdb) n
precompute_hesaff: 2/9> /home/joshuabeard/code/hotspotter/hscom/Parallelize.py(117)_compute_in_serial()
-> result = fn(*args)
(Pdb) args
task_list = [(<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid1_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid1_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid2_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid2_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid3_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid3_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid4_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid4_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid5_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid5_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid6_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid6_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid7_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid7_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid8_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid8_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False})), (<function precompute_hesaff at 0x7ff56892b6e0>, ('/home/joshuabeard/ece177/databases/test0/_hsdb/computed/chips/cid9_CHIP(sz750).png', '/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid9_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz', {'scale_min': 0, 'scale_max': 9001, 'use_adaptive_scale': False}))]
task_lbl = precompute_hesaff: 
verbose = True
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(47)precompute_hesaff()
-> def precompute_hesaff(rchip_fpath, feat_fpath, dict_args):
(Pdb) feat_fpath
'/home/joshuabeard/ece177/databases/test0/_hsdb/computed/feats/cid2_FEAT(hesaff+sift,0_9001)_CHIP(sz750).npz'
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(48)precompute_hesaff()
-> return precompute(rchip_fpath, feat_fpath, dict_args, compute_hesaff)
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(41)precompute()
-> def precompute(rchip_fpath, feat_fpath, dict_args, compute_fn):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(42)precompute()
-> kpts, desc = compute_fn(rchip_fpath, dict_args)
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(88)compute_hesaff()
-> def compute_hesaff(rchip_fpath, dict_args):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(89)compute_hesaff()
-> return detect_kpts(rchip_fpath, dict_args)
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(57)detect_kpts_new()
-> def detect_kpts_new(rchip_fpath, dict_args):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hotspotter/extern_feat.py(58)detect_kpts_new()
-> kpts, desc = pyhesaff.detect_kpts(rchip_fpath, **dict_args)
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(105)detect_kpts()
-> def detect_kpts(img_fpath, use_adaptive_scale=False, **kwargs):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(107)detect_kpts()
-> hesaff_ptr = new_hesaff(img_fpath, **kwargs)
(Pdb) s
--Call--
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(96)new_hesaff()
-> def new_hesaff(img_fpath, **kwargs):
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(98)new_hesaff()
-> hesaff_params = hesaff_param_dict.copy()
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(99)new_hesaff()
-> hesaff_params.update(kwargs)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(100)new_hesaff()
-> hesaff_args = hesaff_params.values()
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(101)new_hesaff()
-> hesaff_ptr = hesaff_lib.new_hesaff_from_params(realpath(img_fpath), *hesaff_args)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(102)new_hesaff()
-> return hesaff_ptr
(Pdb) n
--Return--
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(102)new_hesaff()->70276816
-> return hesaff_ptr
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(109)detect_kpts()
-> nKpts = hesaff_lib.detect(hesaff_ptr)
(Pdb) s
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(111)detect_kpts()
-> kpts = np.empty((nKpts, 5), kpts_dtype)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(112)detect_kpts()
-> desc = np.empty((nKpts, 128), desc_dtype)
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(115)detect_kpts()
-> import pdb
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(116)detect_kpts()
-> pdb.set_trace()
(Pdb) n
> /home/joshuabeard/code/hotspotter/hstpl/extern_feat/pyhesaff.py(118)detect_kpts()
-> hesaff_lib.exportArrays(hesaff_ptr, nKpts, kpts, desc)
(Pdb) s
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(196)from_param()
-> @classmethod
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(198)from_param()
-> if not isinstance(obj, ndarray):
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(200)from_param()
-> if cls._dtype_ is not None \
(Pdb) r
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()-><numpy.c...3bdc2750>
-> return obj.ctypes
(Pdb) a
cls = <class 'numpy.ctypeslib.ndpointer_<f4_2d_ALIGNED_C_CONTIGUOUS_WRITEABLE'>
obj = [[ -8.25391082e-25   4.59023338e-41  -8.25391082e-25   4.59023338e-41
    1.88182698e-36]
 [  0.00000000e+00   1.88182698e-36   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 ..., 
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00]]
(Pdb) n
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(263)get_as_parameter()
-> def get_as_parameter(self):
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()
-> return self._ctypes.c_void_p(self._data)
(Pdb) n
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()->c_void_p(69211808)
-> return self._ctypes.c_void_p(self._data)
(Pdb) n
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(196)from_param()
-> @classmethod
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(198)from_param()
-> if not isinstance(obj, ndarray):
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(200)from_param()
-> if cls._dtype_ is not None \
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(201)from_param()
-> and obj.dtype != cls._dtype_:
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(203)from_param()
-> if cls._ndim_ is not None \
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(204)from_param()
-> and obj.ndim != cls._ndim_:
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(206)from_param()
-> if cls._shape_ is not None \
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(209)from_param()
-> if cls._flags_ is not None \
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(210)from_param()
-> and ((obj.flags.num & cls._flags_) != cls._flags_):
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()
-> return obj.ctypes
(Pdb) n
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/ctypeslib.py(213)from_param()-><numpy.c...3bdc2750>
-> return obj.ctypes
(Pdb) n
--Call--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(263)get_as_parameter()
-> def get_as_parameter(self):
(Pdb) n
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()
-> return self._ctypes.c_void_p(self._data)
(Pdb) n
--Return--
> /home/joshuabeard/.local/lib/python2.7/site-packages/numpy/core/_internal.py(264)get_as_parameter()->c_void_p(69992384)
-> return self._ctypes.c_void_p(self._data)
(Pdb) n
Segmentation fault (core dumped)

Since this is a call to ctypes.c_void_p, I checked the documentation on ctypes. It states:

Note: The code samples in this tutorial use doctest to make sure that they actually work. Since some code samples behave differently under Linux, Windows, or Mac OS X, they contain doctest directives in comments.

Since this is a program meant to be ported from Linux to Windows at some point, I wonder if there is an underlying issue with using some code meant for Windows deployment on a Linux machine...

AudreyBeard commented 7 years ago

At this point, I think we need to reinstall and build the hesaff library

Things to try

mattdioso commented 7 years ago

'> /home/matt/code/matt/hotspotter/hstpl/extern_feat/pyhesaff.py(117)detect_kpts() -> hesaff_lib.exportArrays(hesaff_ptr, nKpts, kpts, desc) (Pdb) hesaff_ptr 67599968 (Pdb) nKpts
750 (Pdb) s --Call-- Bus error (core dumped)'

I got this new error when I tried to step into the exportArrays() function and found this description of Bus Errors:

On most architectures I've used, the distinction is that:

a SEGV is caused when you access memory you're not meant to (e.g., outside of your address space).
a SIGBUS is caused due to alignment issues with the CPU (e.g., trying to read a long from an address which isn't a multiple of 4).
mattdioso commented 7 years ago

HACK

I commented out the line where it calls exportArrays() and it seems to be running normally. Not sure of what impact this has on the program in the long run so this is more of a quick fix and for the sake of a demo.

->#hesaff_lib.exportArrays(hesaff_ptr, nKpts, kpts, desc)

AudreyBeard commented 7 years ago

Notes

hesaff.cpp

To play with hesaff.cpp, make sure you build it using ~/code/hesaff/unix_hesaff_build.sh then copy it to your extern_feats directory with cp ~/code/hesaff/build/libhesaff.so ~/code/hotspotter/hstpl/extern_feat/libhesaff.so

To do all this at once, execute: ~/code/hesaff/unix_hesaff_build.sh; cp ~/code/hesaff/build/libhesaff.so ~/code/hotspotter/hstpl/extern_feat/libhesaff.so

Pickle Problem

rm -r ~/.hotspotter; rm -r ~/code/hotspotter/newdb1/_hsdb/computed/feats

AudreyBeard commented 7 years ago

Updates

4/13/17

AudreyBeard commented 7 years ago

Questions