search

qiskit-community / qiskit-nature

Qiskit Nature is an open-source, quantum computing, framework for solving quantum mechanical natural science problems.
https://qiskit-community.github.io/qiskit-nature/
Apache License 2.0
301 stars 204 forks source link

Fall 2021-Challenge 2 migrate from 0.2.0 to 0.7.1 #1307

Closed calculatechen closed 5 months ago

calculatechen commented 8 months ago

Environment

What is happening?

This is the web link of Challenge 2 ：https://github.com/qiskit-community/ibm-quantum-challenge-fall-2021/blob/main/solutions-by-authors/challenge-2/challenge-2.ipynb

This is the web link of Migration Guides I referenced：https://qiskit.org/ecosystem/nature/migration/0.5_c_electronic_structure.html

I try to migrate from 0.2.0 to 0.7.1, get an error： Can't synchronously write data (file write failed: time = Wed Jan 24 11:27:50 2024 , filename = '/tmp/tmp9f5remmb', file descriptor = 76, errno = 28, error message = 'No space left on device', buf = 0x2ac60e8420a0, total write size = 7065072, bytes this sub-write = 7065072, bytes actually written = 18446744073709551615, offset = 0)

There is a temporary file with a size of about 40g, which is full of space/dev/mapper/CentOS-ROOT

I tried to modify the default path of the TMP folder

os.environ['TMPDIR'] = '/home/user1/scr/tmp/qiskit/tmp'
os.environ['PYSCF_TMPDIR'] = '/home/user1/scr/tmp/qiskit/tmp'

Unfortunately, I got another error： Unable to allocate 64.0 GiB for an array with shape (92665, 92665) and data type float64

How can we reproduce the issue?

The following is my code：

from qiskit_nature.units import DistanceUnit
from qiskit_nature.second_q.drivers import PySCFDriver
from qiskit_nature.second_q.formats.molecule_info import MoleculeInfo
from qiskit_nature.second_q.formats.qcschema_translator import qcschema_to_problem
from qiskit_nature.second_q.problems import ElectronicBasis

# PSPCz molecule
geometry = [['C', [  -0.2316640,    1.1348450,    0.6956120]],
            ['C', [  -0.8886300,    0.3253780,   -0.2344140]],
            ['C', [  -0.1842470,   -0.1935670,   -1.3239330]],
            ['C', [   1.1662930,    0.0801450,   -1.4737160]],
            ['C', [   1.8089230,    0.8832220,   -0.5383540]],
            ['C', [   1.1155860,    1.4218050,    0.5392780]],
            ['S', [   3.5450920,    1.2449890,   -0.7349240]],
            ['O', [   3.8606900,    1.0881590,   -2.1541690]],
            ['C', [   4.3889120,   -0.0620730,    0.1436780]],
            ['O', [   3.8088290,    2.4916780,   -0.0174650]],
            ['C', [   4.6830900,    0.1064460,    1.4918230]],
            ['C', [   5.3364470,   -0.9144080,    2.1705280]],
            ['C', [   5.6895490,   -2.0818670,    1.5007820]],
            ['C', [   5.4000540,   -2.2323130,    0.1481350]],
            ['C', [   4.7467230,   -1.2180160,   -0.5404770]],
            ['N', [  -2.2589180,    0.0399120,   -0.0793330]],
            ['C', [  -2.8394600,   -1.2343990,   -0.1494160]],
            ['C', [  -4.2635450,   -1.0769890,    0.0660760]],
            ['C', [  -4.5212550,    0.2638010,    0.2662190]],
            ['C', [  -3.2669630,    0.9823890,    0.1722720]],
            ['C', [  -2.2678900,   -2.4598950,   -0.3287380]],
            ['C', [  -3.1299420,   -3.6058560,   -0.3236210]],
            ['C', [  -4.5179520,   -3.4797390,   -0.1395160]],
            ['C', [  -5.1056310,   -2.2512990,    0.0536940]],
            ['C', [  -5.7352450,    1.0074800,    0.5140960]],
            ['C', [  -5.6563790,    2.3761270,    0.6274610]],
            ['C', [  -4.4287740,    3.0501460,    0.5083650]],
            ['C', [  -3.2040560,    2.3409470,    0.2746950]],
            ['H', [  -0.7813570,    1.5286610,    1.5426490]],
            ['H', [  -0.7079140,   -0.7911480,   -2.0611600]],
            ['H', [   1.7161320,   -0.2933710,   -2.3302930]],
            ['H', [   1.6308220,    2.0660550,    1.2427990]],
            ['H', [   4.4214900,    1.0345500,    1.9875450]],
            ['H', [   5.5773000,   -0.7951290,    3.2218590]],
            ['H', [   6.2017810,   -2.8762260,    2.0345740]],
            ['H', [   5.6906680,   -3.1381740,   -0.3739110]],
            ['H', [   4.5337010,   -1.3031330,   -1.6001680]],
            ['H', [  -1.1998460,   -2.5827750,   -0.4596910]],
            ['H', [  -2.6937370,   -4.5881470,   -0.4657540]],
            ['H', [  -5.1332290,   -4.3740010,   -0.1501080]],
            ['H', [  -6.1752900,   -2.1516170,    0.1987120]],
            ['H', [  -6.6812260,    0.4853900,    0.6017680]],
            ['H', [  -6.5574610,    2.9529350,    0.8109620]],
            ['H', [  -4.3980410,    4.1305040,    0.5929440]],
            ['H', [  -2.2726630,    2.8838620,    0.1712760]]]

elements = [item[0] for item in geometry]
coordinates = [item[1] for item in geometry]

molecule = MoleculeInfo(elements, coordinates, charge=0, multiplicity=1, units=DistanceUnit.ANGSTROM)
driver = PySCFDriver.from_molecule(molecule=molecule, 
                                   basis='631g*',
                                   )

driver.run_pyscf()
qcschema = driver.to_qcschema()
problem = qcschema_to_problem(qcschema, basis=ElectronicBasis.MO)

What should happen?

When running code, get an error： Can't synchronously write data (file write failed: time = Wed Jan 24 11:27:50 2024 , filename = '/tmp/tmp9f5remmb', file descriptor = 76, errno = 28, error message = 'No space left on device', buf = 0x2ac60e8420a0, total write size = 7065072, bytes this sub-write = 7065072, bytes actually written = 18446744073709551615, offset = 0)

---------------------------------------------------------------------------
ThreadRuntimeError                        Traceback (most recent call last)
File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:475, in half_e1(mol, mo_coeffs, swapfile, intor, aosym, comp, max_memory, ioblk_size, verbose, compact, ao2mopt)
    473 ti0 = log.timer_debug1('gen AO/transform MO [%d/%d]'%(istep+1,nstep), *ti0)
--> 475 async_write(istep, iobuf)
    476 buf2, buf_write = buf_write, buf2

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/lib/misc.py:1076, in call_in_background.__enter__.<locals>.def_async_fn.<locals>.async_fn(*args, **kwargs)
   1075     except Exception as e:
-> 1076         raise ThreadRuntimeError('Error on thread %s:\n%s'
   1077                                  % (self, e))
   1078 handlers[i] = executor.submit(fns[i], *args, **kwargs)

ThreadRuntimeError: Error on thread <pyscf.lib.misc.call_in_background object at 0x2ab853a6acd0>:
[Errno 28] Can't synchronously write data (file write failed: time = Wed Jan 24 11:27:50 2024
, filename = '/tmp/tmp9f5remmb', file descriptor = 76, errno = 28, error message = 'No space left on device', buf = 0x2ac60e8420a0, total write size = 7065072, bytes this sub-write = 7065072, bytes actually written = 18446744073709551615, offset = 0)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/lib/misc.py:1094, in call_in_background.__exit__(self, type, value, traceback)
   1093     else:
-> 1094         handler.result()
   1095 except Exception as e:

File ~/anaconda3/envs/qiskit39/lib/python3.9/concurrent/futures/_base.py:439, in Future.result(self, timeout)
    438 elif self._state == FINISHED:
--> 439     return self.__get_result()
    441 self._condition.wait(timeout)

File ~/anaconda3/envs/qiskit39/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
    390 try:
--> 391     raise self._exception
    392 finally:
    393     # Break a reference cycle with the exception in self._exception

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/lib/misc.py:1074, in call_in_background.__enter__.<locals>.def_async_fn.<locals>.async_fn(*args, **kwargs)
   1073 try:
-> 1074     handlers[i].result()
   1075 except Exception as e:

File ~/anaconda3/envs/qiskit39/lib/python3.9/concurrent/futures/_base.py:446, in Future.result(self, timeout)
    445 elif self._state == FINISHED:
--> 446     return self.__get_result()
    447 else:

File ~/anaconda3/envs/qiskit39/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
    390 try:
--> 391     raise self._exception
    392 finally:
    393     # Break a reference cycle with the exception in self._exception

File ~/anaconda3/envs/qiskit39/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:447, in half_e1.<locals>.save(istep, iobuf)
    446 for icomp in range(comp):
--> 447     _transpose_to_h5g(fswap, '%d/%d'%(icomp,istep), iobuf[icomp],
    448                       e2buflen, None)

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:505, in _transpose_to_h5g(h5group, key, dat, blksize, chunks)
    504 for col0, col1 in prange(0, ncol, blksize):
--> 505     dset[col0:col1] = lib.transpose(dat[:,col0:col1])

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/h5py/_hl/dataset.py:999, in Dataset.__setitem__(self, args, val)
    998 for fspace in selection.broadcast(mshape):
--> 999     self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5d.pyx:282, in h5py.h5d.DatasetID.write()

File h5py/_proxy.pyx:115, in h5py._proxy.dset_rw()

OSError: [Errno 28] Can't synchronously write data (file write failed: time = Wed Jan 24 11:27:50 2024
, filename = '/tmp/tmp9f5remmb', file descriptor = 76, errno = 28, error message = 'No space left on device', buf = 0x2ac60e8420a0, total write size = 7065072, bytes this sub-write = 7065072, bytes actually written = 18446744073709551615, offset = 0)

During handling of the above exception, another exception occurred:

ThreadRuntimeError                        Traceback (most recent call last)
Cell In[3], line 4
      2 from qiskit_nature.second_q.problems import ElectronicBasis
      3 driver.run_pyscf()
----> 4 qcschema = driver.to_qcschema()
      5 problem = qcschema_to_problem(qcschema, basis=ElectronicBasis.MO)
      6 # problem = driver.run()
      7 # print(problem)

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/qiskit_nature/second_q/drivers/pyscfd/pyscfdriver.py:541, in PySCFDriver.to_qcschema(self, include_dipole)
    538     data.hij_mo_b = np.dot(np.dot(data.mo_coeff_b.T, data.hij), data.mo_coeff_b)
    540 data.eri = self._mol.intor("int2e", aosym=8)
--> 541 data.eri_mo = fold(ao2mo.full(self._mol, data.mo_coeff, aosym=4))
    542 if data.mo_coeff_b is not None:
    543     data.eri_mo_bb = fold(ao2mo.full(self._mol, data.mo_coeff_b, aosym=4))

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/__init__.py:159, in full(eri_or_mol, mo_coeff, erifile, dataname, intor, *args, **kwargs)
    156     return mod.full(eri_or_mol, mo_coeff, erifile.name, dataname, intor,
    157                     *args, **kwargs)
    158 else:
--> 159     return mod.full_iofree(eri_or_mol, mo_coeff, intor, *args, **kwargs)

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:588, in full_iofree(mol, mo_coeff, intor, aosym, comp, max_memory, ioblk_size, verbose, compact)
    510 r'''Transfer arbitrary spherical AO integrals to MO integrals for given orbitals
    511 This function is a wrap for :func:`ao2mo.outcore.general`.  It's not really
    512 IO free.  The returned MO integrals are held in memory.  For backward compatibility,
   (...)
    585 (3, 100, 55)
    586 '''
    587 with lib.H5TmpFile() as feri:
--> 588     general(mol, (mo_coeff,)*4, feri, dataname='eri_mo',
    589             intor=intor, aosym=aosym, comp=comp,
    590             max_memory=max_memory, ioblk_size=ioblk_size,
    591             verbose=verbose, compact=compact)
    592     return numpy.asarray(feri['eri_mo'])

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:271, in general(mol, mo_coeffs, erifile, dataname, intor, aosym, comp, max_memory, ioblk_size, verbose, compact)
    269 # transform e1
    270     fswap = lib.H5TmpFile()
--> 271     half_e1(mol, mo_coeffs, fswap, intor, aosym, comp, max_memory, ioblk_size,
    272             log, compact)
    274     time_1pass = log.timer('AO->MO transformation for %s 1 pass'%intor,
    275                            *time_0pass)
    277     def load(icomp, row0, row1, buf):

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:476, in half_e1(mol, mo_coeffs, swapfile, intor, aosym, comp, max_memory, ioblk_size, verbose, compact, ao2mopt)
    473         ti0 = log.timer_debug1('gen AO/transform MO [%d/%d]'%(istep+1,nstep), *ti0)
    475         async_write(istep, iobuf)
--> 476         buf2, buf_write = buf_write, buf2
    478 fswap = None
    479 return swapfile

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/lib/misc.py:1096, in call_in_background.__exit__(self, type, value, traceback)
   1094                 handler.result()
   1095         except Exception as e:
-> 1096             raise ThreadRuntimeError('Error on thread %s:\n%s' % (self, e))
   1098 if self.executor is not None:
   1099     self.executor.shutdown(wait=True)

ThreadRuntimeError: Error on thread <pyscf.lib.misc.call_in_background object at 0x2ab853a6acd0>:
[Errno 28] Can't synchronously write data (file write failed: time = Wed Jan 24 11:27:50 2024
, filename = '/tmp/tmp9f5remmb', file descriptor = 76, errno = 28, error message = 'No space left on device', buf = 0x2ac60e8420a0, total write size = 7065072, bytes this sub-write = 7065072, bytes actually written = 18446744073709551615, offset = 0)

After modifying the TMP path, get another error：

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
Cell In[9], line 5
      2 from qiskit_nature.second_q.problems import ElectronicBasis
      4 driver.run_pyscf()
----> 5 qcschema = driver.to_qcschema()
      6 problem = qcschema_to_problem(qcschema, basis=ElectronicBasis.MO)

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/qiskit_nature/second_q/drivers/pyscfd/pyscfdriver.py:541, in PySCFDriver.to_qcschema(self, include_dipole)
    538     data.hij_mo_b = np.dot(np.dot(data.mo_coeff_b.T, data.hij), data.mo_coeff_b)
    540 data.eri = self._mol.intor("int2e", aosym=8)
--> 541 data.eri_mo = fold(ao2mo.full(self._mol, data.mo_coeff, aosym=4))
    542 if data.mo_coeff_b is not None:
    543     data.eri_mo_bb = fold(ao2mo.full(self._mol, data.mo_coeff_b, aosym=4))

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/__init__.py:159, in full(eri_or_mol, mo_coeff, erifile, dataname, intor, *args, **kwargs)
    156     return mod.full(eri_or_mol, mo_coeff, erifile.name, dataname, intor,
    157                     *args, **kwargs)
    158 else:
--> 159     return mod.full_iofree(eri_or_mol, mo_coeff, intor, *args, **kwargs)

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/pyscf/ao2mo/outcore.py:592, in full_iofree(mol, mo_coeff, intor, aosym, comp, max_memory, ioblk_size, verbose, compact)
    587 with lib.H5TmpFile() as feri:
    588     general(mol, (mo_coeff,)*4, feri, dataname='eri_mo',
    589             intor=intor, aosym=aosym, comp=comp,
    590             max_memory=max_memory, ioblk_size=ioblk_size,
    591             verbose=verbose, compact=compact)
--> 592     return numpy.asarray(feri['eri_mo'])

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File ~/anaconda3/envs/qiskit39/lib/python3.9/site-packages/h5py/_hl/dataset.py:1057, in Dataset.__array__(self, dtype)
   1051 @with_phil
   1052 def __array__(self, dtype=None):
   1053     """ Create a Numpy array containing the whole dataset.  DON'T THINK
   1054     THIS MEANS DATASETS ARE INTERCHANGEABLE WITH ARRAYS.  For one thing,
   1055     you have to read the whole dataset every time this method is called.
   1056     """
-> 1057     arr = numpy.zeros(self.shape, dtype=self.dtype if dtype is None else dtype)
   1059     # Special case for (0,)*-shape datasets
   1060     if self.size == 0:

MemoryError: Unable to allocate 64.0 GiB for an array with shape (92665, 92665) and data type float64

Any suggestions?

No response

mrossinek commented 5 months ago

MemoryError: Unable to allocate 64.0 GiB for an array with shape (92665, 92665) and data type float64

The final error of your message above clearly states that you simply do not have sufficient memory to execute this code on the hardware that you are using. This is unrelated to the code contained in Qiskit Nature.